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1  Executive  Summary  of  the  Phase  I  Effort  Completed  To-Date 

The  Phase  I  approach  could  be  summarized  by  describing  two  independent  but  related  research 
and  development  activities.  One  effort  was  the  creation  of  a  plan  for  the  development  of  a 
Leadership  Develop  Center.  In  the  Phase  I  proposal,  Progeny  Systems  proposed  to  create  a 
simulation  or  scenario  driven  computer-based  assessment  of  an  individual’s  leadership  performance. 
In  order  to  create  this  computer-based  assessment  solution  the  relevant  literature  had  to  be  reviewed 
to  develop  the  theoretical  framework  for  the  proof  of  concept  solution.  Progeny  Systems  reviewed 
literature  on  feedback  interventions,  naturalistic  decision  making,  leadership  behaviors  and  the 
leadership  literature  in  general. 

The  other  significant  effort  was  the  creation  of  a  system  to  deliver  content,  capture  user 
responses,  and  report  the  results  out  to  an  external  system.  This  Simulation  Delivery  System  was 
created  using  XML,  web  services  and  the  .Net  2.0  framework  to  minimize  the  client-side  code  and 
keep  as  much  processing  and  functionality  on  the  web  server  as  possible.  Moving  the  functionality 
out  of  the  Simulation  Environment  also  granted  flexibility  in  which  Simulation  Engine  could  be 
used.  These  web  services  were  further  split  into  three  different,  but  complementary  functions.  Get 
Leadership  Data,  Get  Simulation  Parameters,  and  Store  Leadership  Simulation  Results.  A  detailed 
description  of  the  Phase  I  work  is  described  in  Section  3 
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Commercialization 

Value  Added  Opportunities 

•  Computer-Based  Simulation  Examinations  and  Assessments 

•  Improve  Prediction  and  Quantification 

•  Provide  Multiple  DoD  Agencies  With  Improved  Assessment 
Models 

Employee  Selection 

•  Selection  System  To  Predict  Success  In  Leadership 
Domains  (Managers,  Executives) 

Performance  Appraisal 

•  Performance  In  Leadership  Dimensions 


Qualifications 

Progeny  Corporate  Competence 

•  Improving  Quality  Of  Life  Though  Technologies  And  Improved  Processes 

•  Innovative  Use  Of  Cross-Domain  Expertise 

•  Taking  Innovative  Concepts  and  Delivering  Them  to  War  Fighters 
Hz  Assessments  Competence 

•  Design  Selection  Systems 

•  Develop  New  Selection  Protocols 

•  Applied  Advanced  Statistical  Procedures 
Principal  Investigators 

•  Doctorates  In  Systems  Engineering  And  Industrial  Psychology 

•  Internationally  Recognized  Psychometric  Expert 
Similar  Research  and  Development 

•  Selection  System  For  Harbor  Pilots 

•  Assessment  Systems  For  Social  Workers 

•  Assessing  “Hybrid"  Sailors 

•  Manning  Reduction  Through  Personnel  Selection  and  Task  Automation 

•  Fonward  Deployed  Learning  Management  Systems 


2  Leadership  Assessment  Center 

While  the  operating  theater,  the  details  of  the  conflict,  and  tactics  of  the  enemy  often  change 
from  engagement  to  engagement  or  even  during  the  same  conflict,  the  value  of  finding,  assessing 
and  retaining  leaders  is  always  a  major  task  for  the  Army.  Historically,  it  has  been  difficult  for  the 
Army  to  identify  individuals  with  high  leadership  potential  a  priori  to  some  critical  event.  In  other 
words,  the  nature  of  a  battlefield  often  leads  to  leaders  being  identified  based  upon  their 
performance  during  the  heat  of  the  battle.  However,  waiting  until  a  conflict  arises  is  not  the  most 


1 


efficient  or  safest  way  for  leaders  to  be  identified.  The  effort  described  in  this  final  report  represents 
one  method  of  developing  a  Leadership  Assessment  Center  based  upon  both  existing  technologies 
and  assessment  batteries  and  the  extension  of  technologies  not  normally  associated  with  Leadership 
assessment.  Through  the  continued  development  of  the  prototype  completed  for  this  Phase  I  and 
the  extension  on  the  relevant  literature,  Progeny  Systems  is  confident  that  an  end  to  end  solution  for 
a  Leadership  Assessment  Center  can  be  developed. 


End-to-End  Solution  Requirements 


Ability  To  Establish  A  Theoretical 
Foundation  For  the  Leadership 
Construct. 


Capitalize  on  Existing  Assessment 
Techniques  and  Validated  Assessment 
Batteries  for  Component  Dimensions  of 
the  Leadership  Construct 

Ability  To  Move  Data  and  Information 
From  Central  information  warehouses 
to  Distributed  Locations 


Ability  to  Develop  A  Software  Based 
Leadership  Assessment  Centers 


Ability  to  Get  the  Solution  To  The 
Warfighter 


Progeny  Efforts  That  Support  Vision 


»  Progeny  Systems  and  it’s  teammates  have  the  technical  and 
research  skills  necessary  to  identify  the  relevant  academic 
and  industrial  literature  to  create  a  working,  testable  and 
extensible  definition  of  the  Leadership  Construct 

•  Through  the  use  of  commercial  Learning  Management 
Systems,  Progeny  Systems  already  has  the  expertise 
necessary  to  combine  existing  multi-modal  assessment 
techniques  into  a  single  seamless  Leadership  Assessment 
Center 

>  In  order  to  capitalize  on  the  investment  necessary  to  create 
and  maintain  different  assessment  instruments,  Progeny 
Systems  proposes  to  use  its  proven  technology  to  manage 
the  data,  ensure  its  credibility  /  authenticity,  deliver  the 
content  to  the  end  user  and  finally  report  the  results  to  the 
Army  for  integration  into  its  personnel  information 
systems. 

'  Progeny  Systems  has  developed  portions  of  the  software 
functionality  within  the  Non- tactical  Data  Processing 
System  (NTDPS)  for  the  Navy  and  for  many  other 
programs  for  all  branches  of  the  Armed  Services.  Progeny 
Systems  has  the  Psychometric  expertise  to  build  a 
Leadership  Assessment  Center  that  will  provide  the  Army 
with  unparallel  ability  to  assess  individuals  on  the 
Leadership  Construct.  Progeny  Systems  is  totally 
committed  to  working  with  third  party  companies  whose 
functionality,  technology,  or  assessment  batteries  the 
Government  wants  integrated  into  the  solution.  Progeny 
Systems  is  a  member  of  the  NAVSEA  Open 
Architecture/ Open  Source  (OA)  initiative  as  well  as  other 
industry  standards  and  committed  to  their  principles  of 
open  collaboration. 

Progeny  Systems  has  fielded  The  NTDPS  application  suite 
on  VIRGINIA  and  15  688  class  submarines.  Progeny 
Systems  is  supporting  NSWC  Crane  in  the  fielding  of 
Distance  Support  1.0  on  surface  ships  starting  in  the 
Summer  of  2006.  While  Progeny  Systems  has  yet  to 
deliver  a  system  to  the  Army,  our  proven  track  record  with 
getting  systems  to  warfighters  in  the  Navy  shows  that  we 
understand  the  issues  facing  the  introduction  of 
technology  to  the  Armed  Services 
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3  Simulation  Delivery  System  Development 

The  primaiy  goal  of  for  this  component  of  the  Phase  I  effort  was  to  develop  a  system  for 
delivering  customi2able  simulations  to  the  Learner.  The  simulations  needed  to  be  aware  of  the 
Learner’s  existing  skill  set  (i.e.  education,  training,  relevant  experience)  as  well  as  any  parameters 
defined  by  the  Training  Administrator  to  either  increase  or  decrease  overall  system  complexity. 
Progeny  Systems  started  this  effort  by  defining  the  various  Data  Types  that  would  be  required  for 
tracking  the  learners  skill  set  and  for  defining  “complexity”  in  terms  of  a  leadership  assessment 
center.  Next,  Progeny  Systems  defined  and  developed  the  Data  Access  methods  to  provide  a 
standard  interface  between  the  Learner’s  data  and  the  target  Simulation  Environment.  Progeny 
Systems  then  developed  the  tools  required  to  manipulate  the  stored  Learner  data.  Some  of  which 
will  be  replaced  by  existing  Army  tools,  or  other  commercial  implementations  in  the  Phase  II  effort. 
Finally,  Progeny  Systems  designed  and  developed  a  prototype  Simulation  Engine  to  demonstrate  the 
theoretical  concepts  and  to  verify  and  validate  the  logistics  of  the  data  movement,  capture  and 
storing  processes, 

3.1.1  Data  Types 

To  ease  the  deployment  requirements  and  provide  greater  flexibility,  all  of  the  Data  Types  are 
stored  in  XML.  Associated  schemas  have  also  been  created  for  validating  the  stored  Data  Types. 

3.1. 1.1  Leadership  Training  Records 

Leadership  Training  Records  (LTRs)  are  a  subset  of  the  Soldier’s  complete  Electronic  Training 
Jacket.  LTRs  represent  a  Learner’s  existing  demonstrated  Leadership  Skills  across  all  the  identified 
dimensions  of  leadership  in  a  manner  similar  to  current  Learner  Management  System’s  (LMS) 
learned  skill  format.  (Progeny  acknowledges  that  “LTRs”  do  not  exist  at  this  moment  in  a  quantified 
form,  but  are  being  used  for  this  prototype  to  facilitate  development  and  simply  the  mechanics  of 
transitioning  from  a  proof  of  concept  to  a  prototype.  The  effort  needed  to  construct  the  data  and 
measurement  instruments  for  the  LTR  concept  is  described  in  more  detail  in  the  Phase  II  work 
plan.)  This  design  approach  was  chosen  to  ease  the  process  of  integrating  the  LTRs  into  the  existing 
Army  LMS  during  the  Phase  II  effort. 

3.1. 1.2  Simulation  Adjustment  Parameters 

Simulation  Adjustment  Parameters  (SAPs)  were  developed  to  provide  a  mechanism  for 
Training  Administrators  to  customize  the  simulation  difficulty  and  other  simulation  specific 
parameters.  SAPs  are  created  on  a  per  Simulation  basis  and  contain  details  about  all  of  the 
adjustable  variables  that  the  Simulation  uses.  Details  on  how  to  present  the  parameter  to  the 
Training  Administrator  are  also  stored  in  the  SAP.  The  current  implementation  supports  displaying 
Strings,  Integers,  Floats  (Double),  Boolean  and  Ranges.  In  the  future  any  data  type  that  can  be 
parsed  from  a  String  can  be  utilized.  Additionally,  a  detailed  description  about  what  each  parameter 
controls  is  included  to  help  the  Training  Administrator  properly  adjust  the  simulation  complexity. 

3.1. 1.3  Individualized  Simulation  Adjustment  Parameters 

Individualized  Simulation  Adjustment  Parameters  (ISAPs)  contain  the  adjusted  parameter 
values  for  a  specific  combination  of  Simulation  and  Learner.  The  parameters  stored  in  the  ISAP  are 
based  off  of  the  parameters  defined  in  the  associated  Simulation’s  SAP  (see  section  3. 1.1. 2).  In  the 
current  prototype  implementation  there  can  only  be  one  ISAP  associated  per  Learner  and 
Simulation  combination.  The  Training  Administrator  must  redefine  the  parameters  between  each 
simulation  attempt  by  the  Learner  if  he/ she  wishes  to  adjust  the  difficulty. 
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3.1. 1.4  Leadership  Simulation  Resuits 

Leadership  Simulation  Results  are  expected  to  be  generated  by  the  Simulation  Engine.  They 
contain  three  main  sections: 

Assessment  Scores 

The  Assessment  Scores  section  contains  the  objective  assessment  results  for  the  Leadership 
Skills  or  Behaviors  measured  in  the  Simulation.  These  results  utilize  the  Leadership  Training 
Records  (see  section  3. 1.1.1)  Leadership  Skill  format  for  easy  parsing  into  the  Learner’s  LTR. 

Learner  Activity  Log 

The  Learner  Activity  Log  is  a  record  of  actions  taken  by  the  Learner  during  the  simulation. 
This  record  is  intended  to  be  used  during  Simulation  Performance  feedback  interventions  and 
debriefs. 

Individualized  Simulation  Adjustment  Parameters 

The  Leadership  Simulation  Results  contain  a  copy  of  the  ISAP  values  for  reference  purposes. 
This  will  allow  the  Training  Administrator  to  see  the  effect  of  the  various  ISAP  parameter  values  on 
the  Learner’s  training  performance. 

3.1. 1.5  Prototype  Simulation  Script 

To  showcase  the  Phase  I  innovations  Progeny  Systems  needed  to  develop  a  basic  Simulation 
Engine.  Part  of  that  development  included  developing  a  reusable  format  for  defining  the  Simulation 
Events.  Progeny  Systems  chose  to  implement  a  rudimentaiy  Decision  Tree  similar  to  a  “Qioose 
your  own  ending”  story.  As  the  solution  matures,  a  more  complicated  Bayesian  approach  could  be 
utilized  on  the  same  framework  as  the  currently  implemented  deterministic  decision  tree.  Through 
the  use  of  Bayesian  probability  theory,  decisions  can  take  a  more  varied  approach  with  the  added 
realism  that  the  same  decision  every  time  through  the  system  might  not  produce  the  same  results. 
Just  as  in  interacting  with  people,  one  technique  might  be  appropriate  one  day,  but  then  a  different 
technique  would  need  to  be  used  on  a  second  incident  even  through  all  the  relevant  characteristics 
of  the  situation  appear  to  be  the  same.  The  Decision  Tree  consists  of  various  Node  types: 

Forced  Continue  Node 

A  Forced  Continue  Node  is  used  to  progress  the  simulation  story  without  presenting  the  learner 
with  a  decision.  At  the  end  of  the  node  text  the  user  is  presented  with  a  button  that  links  them  to 
the  next  Node  in  the  stoiy. 

Conditional  Redirect  Node 

A  Conditional  Redirect  Node  is  used  when  the  Simulation  Author  wants  to  introduce  logic 
based  branching  into  the  simulation  story.  When  the  Simulation  Engine  encounters  a  Conditional 
Redirect  Node  it  compares  the  value  of  the  specified  Conditional  Variable  to  the  available  options 
and  redirects  to  the  specified  Node  upon  a  match.  Linking  several  Conditional  Redirect  Nodes 
allows  the  Simulation  Author  to  make  very  complex  decisions  with  little  or  no  programming  skill. 

Decision  Node 

The  Decision  Node  type  is  the  primary  type  of  node  used.  It  contains  text  for  the  simulation 
story,  a  question  to  the  Learner  about  what  action  they  wish  to  take  and  a  Hst  of  available  options. 
Each  option  contains  a  reference  to  the  Node  that  should  be  loaded  if  it  is  chosen.  Options  may 
also  modify  simulation  variable  values  when  selected.  Additionally,  the  presented  options  can  be 
limited  based  on  existing  variable  values.  This  ability  is  utilized  when  presenting  more  options  to  an 
experienced  Learner  and  fewer  options  to  the  novice  Learner. 
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3.1.2  Data  Access  Web  Services 

During  the  Phase  I  Simulation  Engine  evaluations  Progeny  Systems  determined  that  Web 
Services  could  be  utilized  to  deliver  the  Leadership  Training  Records  and  Simulation  Adjustment 
Parameters  to  almost  any  Simulation  Engine.  By  defining  standard  Data  Access  Web  Services 
Progeny  could  remove  the  burden  of  getting  and  saving  Leadership  Skills  from  the  Simulation 
Author.  The  Simulation  Author  would  also  be  able  to  choose  the  Simulation  Engine  that  best 
displayed  their  Simulation  instead  of  being  forced  to  use  a  standard  Simulation  Engine. 

As  part  of  the  Prototype  Simulation  Engine  development  that  Progeny  Systems  performed  to 
showcase  the  Phase  I  innovations,  Progeny  Systems  also  developed  the  Prototype  Get  Simulation 
Web  Service.  This  service  is  not  expected  to  be  used  by  other  Simulation  Engines. 

3.1. 2.1  Get  Leadership  Data 

The  Get  Leadership  Data  web  service  provides  a  standard  interface  for  retrieving  a  Learner’s 
Leadership  Training  Records  (see  section  3. 1.1.1).  The  Simulation  Engine  is  expected  to  call  the 
Get  Leadership  Data  web  service  passing  the  Learner’s  Unique  ID.  If  an  LTR  is  found  for  the 
specified  Learner  the  Get  Leadership  Data  web  service  will  remm  an  XML  formatted  LTR. 
Otherwise,  it  will  remm  an  empty  XML  element.  The  Simulation  Engine  can  then  utilize  the 
Learner  Skills  in  whatever  way  the  Simulation  Author  intended. 

3.1. 2.2  Get  Simulation  Parameters 

The  Get  Simulation  Parameters  web  service  provides  a  standard  interface  for  retrieving 
Individualized  Simulation  Adjustment  Parameters  (see  section  3. 1.1. 2)  for  a  given  Simulation  and 
Learner  combination.  The  Simulation  Engine  is  expected  to  call  the  Get  Simulation  Parameters  web 
service  passing  the  Learner’s  Unique  ID  and  the  Simulation’s  Unique  ID.  If  an  ISAP  is  found  for 
the  specified  Learner  and  Simulation  combination  the  Get  Simulation  Parameters  web  service  will 
return  an  XML  formatted  ISAP.  Ckherwise,  it  will  return  an  empty  XML  element.  The  Simulation 
Engine  can  easily  utilize  the  parameters  in  the  ISAP  to  set  the  default  state  of  the  Simulation 
Variables. 

3.1. 2.3  Store  Leadership  Simulation  Results 

The  Store  Leadership  Simulation  Results  web  service  provides  a  standard  interface  for  saving  a 
Learner’s  Leadership  Simulation  Results  (see  section  3. 1.1.4).  The  Simulation  Engine  is  expected  to 
call  the  Store  Leadership  Simulation  Results  web  service  passing  the  Leadership  Simulation  Results 
for  the  completed  Simulation.  The  Leadership  Simulation  Results  are  made  up  of  Performance 
Scores,  the  Learner’s  Activity  Log  and  a  record  of  the  ISAP  used  when  launching  the  Simulation. 
The  Simulation  Author  is  required  to  ensure  that  the  Simulation  Engine  he  or  she  chooses  can 
utilize  the  Leadership  Simulation  Result  format. 

3.1. 2.4  Prototype  Get  Simulation 

To  ease  the  Phase  I  Prototype  Simulation  Engine  deployment.  Progeny  Systems  chose  to  utilize 
a  web  service  architecture  for  retrieving  the  Simulation  Script  (see  section  3. 1.1. 5)  and  Default 
Simulation  Adjustment  Parameters  (see  section  3. 1.1.2).  The  Prototype  Get  Simulation  web  service 
provides  a  standard  interface  for  accessing  the  Simulation  Script  and  the  Simulation  Adjustment 
Parameters  used  for  setting  the  default  Simulation  Variable  values.  The  Simulation  Engine  is 
expected  to  call  the  Prototype  Get  Simulation  web  service  passing  the  Simulation’s  Unique  ID.  If 
the  Simulation  Script  is  found  for  the  specified  Simulation  ID  the  Prototype  Get  Simulation  web 
service  returns  the  XML  formatted  Simulation  Script  and  Simulation  Adjustment  Parameters. 
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The  Prototype  Get  Simulation  web  service  is  only  intended  to  be  utilized  by  the  Phase  I 
Prototype  Simulation  Engine.  However,  nothing  prevents  other  Simulation  Engines  from  accessing 
it  if  desired. 

3.1.3  Simulation  Environment 

To  showcase  the  Data  Types  and  the  standard  Data  Access  web  services  Progeny  Systems 
developed  a  Prototype  Simulation  Environment.  Some  sections  of  this  Simulation  Environment 
were  modeled  off  of  existing  Army  technology  and  are  intended  to  be  replaced  with  the  Army 
technologies  or  other  commercially  available  technologies  during  the  Phase  II  integration  efforts. 
Some  of  the  functionality  developed  during  this  effort  is  only  intended  to  give  the  Learner  or 
Training  Administrator  the  general  idea  of  what  is  expected  to  occur  and  is  not  entirely 
representative  of  how  the  prototype  system  would  look  or  function. 

3.1. 3.1  Login  and  Simulation  Selection 

Normally  the  Login  and  Simulation  Selection  functionality  will  be  handled  by  the  Army  Learner 
Management  System  (LMS).  Since  integrating  the  Simulation  Environment  into  the  existing  Army 
LMS  was  beyond  the  scope  of  the  Phase  I  initiative,  Progeny  developed  a  basic  login  process.  The 
login  process  is  based  off  of  the  Navy  Knowledge  Online  login  method.  The  Learner  is  required  to 
enter  their  Social  Security  Number,  their  Last  Name  and  their  Date  of  Birth.  If  aU  of  these  fields 
match  the  record  stored  in  the  Learner  Repository  the  Learner  is  signed  in  to  the  Phase  I  Basic  LMS 
(PI-BLMS).  The  Learner  is  redirected  to  the  Simulation  Selection  screen  of  the  PI-BLMS  and 
prompted  to  select  one  of  the  available  Simulations  to  launch.  This  process  would  normally  be 
handled  by  the  Army  LMS  Course  Selection  feature  and  is  intended  to  be  replaced  during  the  Phase 
II  LMS  integration  effort. 


3.1. 3.2  Simulation  Engine  -  Load  Simulation 


Figure  1  Data  Movement  -  Simulation  Start 


Figure  1  depicts  the  flow  of  data  utilized  by  the  Prototype  Simulation  Engine.  With  the 
exception  of  the  Get  Simulation  process  marked  as  Prototype  Only,  Figure  accurately  represents 
the  expected  data  flow  for  any  Simulation  Engine  utilized  by  Simulation  Authors. 

The  Prototype  Simulation  Engine  is  passed  the  Learner’s  Unique  ID  and  the  Simulation’s 
Unique  ID  by  the  PI-BLMS.  The  Simulation  starts  by  calling  the  Prototype  Get  Simulation  web 
service  (see  section  3. 1.2.4)  to  retrieve  the  selected  Simulation  and  associated  Simulation  Adjustment 
Parameters.  If  the  Prototype  Get  Simulation  web  service  returns  an  empty  XML  element,  the 
Simulation  Engine  issues  a  “Simulation  Not  Found”  fatal  error  and  exits.  Otherwise,  the  Simulation 
Engine  loads  the  Simulation  Variables  defined  in  the  Simulation  Script.  Next,  the  Simulation 
Engine  loads  the  default  values  stored  in  the  Simulation  Adjustment  Parameters  into  the  Simulation 
Variables.  Then  the  Simulation  Engine  initiates  a  request  to  the  Get  Leadership  Data  web  service 
(see  section  3. 1.2.1).  After  receiving  the  Leadership  Training  Records  for  the  specified  Learner,  the 
Simulation  Engine  loads  the  Leadership  Skill  values  into  the  associated  Simulation  Variables  if  they 
exist.  If  a  corresponding  Simulation  Variable  doesn’t  exist  for  the  Learner’s  Leadership  Skill  it  is 
ignored  since  the  Simulation  is  not  designed  to  test  or  utilize  that  specific  skill.  Once  the  Learner’s 
existing  Leadership  Skills  are  loaded  the  Simulation  Engine  calls  the  Get  Simulation  Parameters  web 
service  (see  section  3. 1.2.2)  passing  the  Learner’s  Unique  ID  and  Simulation’s  Unique  ID.  If  the 
Get  Simulation  Parameters  web  service  returns  Individualized  Simulation  Adjustment  Parameters 
the  Simulation  Engine  loads  the  values  stored  in  the  ISAP  into  the  Simulation  Variables.  Finally,  the 
Simulation  Engine  executes  any  other  Load  time  code  specified  by  the  Simulation  Author. 

3.1. 3.3  Simulation  Engine  >  Execute  Script 

The  Prototype  Simulation  Engine  reads  the  Root  Node  from  the  Simulation  Script  and  passes  it 
to  the  Node  Processing  Engine.  The  Node  Processing  Engine  determines  the  Node  Type  and 
handles  its  contents  accordingly. 

Forced  Continue  Node 

To  handle  a  Forced  Continue  Node  (see  section  3. 1.1.5)  the  Node  Processing  Engine  displays 
the  Node  Text  and  generates  a  “Continue”  button  that  references  the  specified  destination  Node. 

Conditional  Redirect  Node 

To  handle  a  Conditional  Redirect  Node  (see  section  3. 1.1.5)  the  Node  Processing  Engine 
compares  the  value  of  the  Conditional  Variable  to  the  available  Redirect  Paths.  If  the  value  matches 
one  of  the  Redirect  Paths  then  the  Current  Node  is  set  to  the  Redirect  Path’s  Destination  Node  and 
the  Node  Processing  Process  repeats. 

Decision  Node 

To  handle  a  Decision  Node  (see  section  3. 1.1.5)  the  Node  Processing  Engine  displays  the  Node 
Text  and  the  Node  Question.  The  Node  Processing  Engine  then  begins  to  populate  the  available 
actions  list  by  evaluating  any  Option  Requirements.  When  evaluating  Option  Requirements  the 
Node  Processing  Engine  compares  the  Option  Requirement  Variable  value  to  the  matching 
Simulation  Variable  value.  If  the  values  match  then  the  Option  is  added  to  the  available  actions  list. 

By  selecting  an  Option  or  pressing  a  “Continue”  button  the  Learner  modifies  the  Current  Node 
and  the  Learner  Activity  Log  is  updated.  In  the  case  of  selecting  an  Option,  Simulation  Variable 
values  may  be  modified.  The  Simulation  Engine  then  repeats  the  Node  Processing  process  until  it 
finally  reaches  the  End  Node. 
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Figure  2  Data  Movement  -  Simulation  End 


Figure  2  represents  the  data  flow  once  the  Simulation  Engine  reaches  the  Simulation  End.  In 
the  Phase  I  Prototype  Simulation  Engine  the  Simulation  End  is  achieved  once  the  End  Node  is 
rendered.  The  Simulation  Engine  then  processes  the  Performance  and  generates  scores  for  the 
tested  Leadership  Skills.  This  data  is  then  packaged  with  the  Learner  Activity  Log  and  the  initial 
Individualized  Simulation  Adjustment  Parameters.  Finally,  the  Simulation  Engine  calls  the  Store 
Leadership  Simulation  Results  web  service  (see  section  3. 1.2.3)  and  passes  the  compiled  Leadership 
Simulation  Result  Data. 

In  the  Phase  I  Prototype  this  data  must  be  manually  loaded  into  the  Leadership  Training 
Records  for  the  Learners.  Normally  this  process  would  be  handled  by  an  LMS  and  it  should  be 
automated  during  the  Phase  II  LMS  integration  effort. 

3.1.4  Data  Manipulation  Tools 

For  the  Phase  I  effort  Progeny  developed  3  primary  tools  for  working  with  the  Data  Types  used 
by  the  Prototype  Simulation  Environment.  Only  the  Individual  Leadership  Simulation  Adjustment 
Tool  was  expected  to  be  a  deliverable.  The  other  two  will  either  be  replaced  by  existing  Army  or 
commercial  technologies  during  the  Phase  II  effort  or  simply  discarded  if  they  are  no  longer  needed. 
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3.1. 4.1  Individual  Leadership  Simulation  Adjustment  Tool 

<r'  ■  ■■  '  ,  ■  ■  ■  1-1..  .  .  ■■■  ,  .  . 
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Figure  3  Individual  Learner  Simulation  Adjustment  Tooi 


Figure  3  shows  the  Individual  Leadership  Simulation  Adjustment  Tool  (TLSAT)  Prototype. 
This  tool  is  designed  to  generate  the  Individualized  Simulation  Adjustment  Parameters  (see  section 
3. 1.1.3)  for  use  in  the  Simulation  Engine.  The  ILSAT  Prototype  is  one  example  of  how  the  ILSAT 
could  be  implemented.  It  was  designed  as  a  proof  of  concept  and  determined  that  most  functional 
requirements  that  the  Army  may  have  regarding  system  requirements  could  be  implemented  at  a 
later  date.  This  version  requires  the  .Net  2.0  framework  to  be  installed  on  the  client  machine  and 
utilizes  web  services  to  store  the  generated  ISAP  on  the  Prototype  Simulation  Environment  server. 
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Due  to  the  open  design  of  the  ISAP  format,  the  ILSAT  can  eventually  be  developed  in  any 
Army  system  requirements  compliant  environment.  The  ILSAT  implementation  technology  would 
simply  be  required  to  understand  the  Simulation  Adjustment  Parameters  (see  section  3. 1.1. 2)  to 
display  the  appropriate  form  controls  and  be  able  to  generate  the  ISAP  formatted  results.  With 
these  minimal  requirements  the  ILSAT  could  be  easily  provided  as  a  web  application  or  imbedded 
into  the  LMS. 

3.1.4.2  Prototype  Learner  Creation  and  Progress  Tracking  Tooi 

The  Learner  Creation  and  Progress  Tracking  Tool  (LCPTT)  was  developed  to  provide  an 
administrative  interface  for  the  Phase  I  Basic  LMS  (PI-BLMS).  This  interface  gives  administrators 
the  ability  to  create,  modify  and  delete  Learners  in  the  system.  It  also  provides  a  method  for 
creating,  modifying  and  deleting  existing  Leadership  Skills  for  each  user.  This  tool  is  intended  to  be 
replaced  by  the  Army  LMS  during  the  Phase  II  LMS  integration  effort. 


Prototype  Simulation  Script  Builder 
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Figure  4  Prototype  Simulation  Script  Builder 

Figure  4  shows  the  Decision  Tree  tab  of  the  Prototype  Simulation  Script  Builder.  While 
developing  sample  Simulation  Scripts,  Progeny  discovered  that  it  was  very  tedious  and  error  prone 
to  write  the  required  XML  by  hand.  Progeny  solved  this  problem  by  developing  the  Prototype 
Simulation  Script  Builder.  This  tool  allows  the  Simulation  Author  to  define  all  of  the  Simulation 
Variables  and  Decision  Tree  Nodes  in  a  rapid  and  easily  understandable  manner.  Simulation 
Adjustment  Parameters  are  also  generated  based  off  of  parameters  set  when  creating  Simulation 
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Variables.  This  tool  will  either  be  altered  or  abandoned,  depending  on  the  use  of  Simulation  Scripts 
in  the  Phase  II  effort. 

3.2  Research  into  Gaming  and  Simulation  Technologies 

See  Appendix  B 

Part  of  Progeny's  original  Phase  I  proposal  included  the  development  of  a  simulation  or  gaming 
environment  to  support  the  rapid  prototyping  of  scenarios  for  the  purpose  of  leadership  assessment. 
Through  discussions  with  the  sponsor,  this  effort  was  redirected  to  the  research  and  effort  described 
above.  However,  since  the  phase  II  effort  may  include  the  development  of  a  more  immersive 
assessment  environment,  the  evaluation  and  results  of  the  gaming  engines  is  included. 

The  results  for  the  Phase  I  prototype  effort  pointed  to  the  use  of  Macromedia  Flash.  "While  not  as 
easy  to  develop  simulations  with  initially  as  some  of  the  other  engines  Progeny  intended  to  release 
the  code  that  powers  the  prototype  simulation.  This  code  can  be  utilized  by  future  simulations  to 
reduce  development  time.  The  primary  reason  why  Progeny  chose  to  utilize  Flash  for  the  Phase  I 
prototype  is  its  ability  to  be  deployed  over  the  web.  Progeny  will  not  have  to  develop  extensive 
code  to  deliver  Flash  simulations  through  existing  LCMS  software  such  as  OutStart’s  Evolution. 
This  will  greatly  reduce  the  work  needed  to  build  an  effective  training  simulation  prototype.  Since 
the  gaming  development  aspect  of  the  phase  I  effort  was  deprecated,  the  analysis  as  to  the  preferred 
gaming  engine  will  be  revisited  if  Progeny  is  awarded  a  Phase  II. 


4  Theoretical  Framework 

The  other  major  effort  of  this  phase  I  project  has  been  a  review  of  relevant  literature  to  further 
refine  the  technical  efforts  as  well  as  to  create  a  theoretical  foundation  for  the  Phase  II  objectives. 
The  research  primarily  focused  on  reviewing  the  literature  in  the  Leadership  domain  (both  military 
and  pubHc  sector).  It  is  important  to  note  an  underlying  assumption  of  the  leadership  constmct  is 
that  leadership  is  multi-dimensional.  Figure  5  presents  a  method  similar  to  the  Nav/s  5  Vector 
Model  to  show  how  an  individual  could  be  assessed  across  a  multi- dimensional  constmct  like 
Leadership.  An  individual  could  be  rated  anywhere  on  the  continuum  of  multiple  independent 
dimensions  {note:  independent  nmmng  that  the  dinvnsims 
are  assessed  isith  different  instruments,  not  necessarily  that  the 
cUrrensions  are  not  correlated  in  some  that  when  taken 
together  would  allow  for  an  assessment  of  the  whole 
multi-dimensional  constmct.  The  relationships 
between  the  dimensions  would  not  have  to  be  fixed, 
rather  certain  dimensions  could  be  weighted  higher 
based  upon  their  predictive  capability  or  overall 
importance.  The  methodology  also  supports  the 
potential  of  non-linear  interactions  or  combinations  of 
the  dimensions.  In  other  words  a  “High”  score  on  A, 
and  B  and  a  low  score  on  B,  C  and  D  might  lead  to  an 
overall  assessment  greater  than  a  “Medium”  score  on 
all  dimensions.  Through  the  utilization  of  this  sort  of 
methodology.  Progeny  systems  can  develop  the  overall 
leadership  assessment  center  in  a  phased  approach  as 
well  as  take  advantage  of  existing  research.  To  put 
differently,  the  sponsor  can  add  or  remove  Dimensions  as  funding,  available  research  or  needs 
change  without  affecting  the  existing  dimensions  in  the  Leadership  Assessment  Center. 


ST 


Fiaure  5;  Multidimensional  view  of  the  Leadershio  Construct 
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Another  underlying  assumption  of  the  Leadership  constmct  is  that  it  can  be  modeled  in  some 
sort  of  continuous  distribution.  To  state  differently,  Leadership  is  not  a  construct  that  a  person 
either  has  or  does  not  have  (binary  distribution),  but  rather  people  have  different  levels  of 
leadership.  While  the  distribution  of  the  leadership  ability  may  not  follow  the  normal  distribution 
pattern,  Figure  6  represents  a  nominal  curve  by  which  leadership  may  exist.  The  continuous 
distribution  assumption  leads  to  a  few  difficulties  that  while  out  of  the  scope  of  this  proposal  are 
interesting  to  note.  One  difficulty  arises  from  if  one  wishes  to  use  the  leadership  construct  for  job 
selection.  It  would  seem  that  if  one  could  argue  that  person  A  is  objectively  a  better  leader  than 
person  B  by  some  measured  amount,  then  person  A  would  be  a  better  candidate.  However,  both 
candidates  could  surpass  the  threshold  for  the  necessary  leadership  for  a  job.  In  this  situation,  the 
job  selection  should  depend  on  some  other  measurement,  but  without  the  ability  to  create 
thresholds  of  perfomance  needed  for  certain  jobs  or  taks,  a  validated  Leadership  measure  might 
create  new  problems. 

The  last  assumption  Progeny  made  with  regard  to  the  leadership  contruct  was  that  leadership 
performance  can  be  improved  via  training  and  external  aids.  This  last  assumption  is  important 
because  if  leadership  performance  was  fixed  then  tme  thresholds  of  performance  could  be  set.  To 
put  differently,  one  could  assess  individuals  along  the  leadership  constmct  and  if  they  didn’t  surpass 
a  specific  “cut  score”  then  they  would  be  ineligible  for  a  certain  job.  However,  since  our 

assumption  is  that  leadership 
performance  can  change  over 
time,  a  primary  benefit  for 
measuring  leadership 

performance  is  not  so  much  as 
a  selection  criterion,  but  rather 
as  a  training  aide  like  learning 
tactics  or  measureing  of 
performance  for  weapons 
handeling.  There  would  be 
some  utility  for  job  selection, 
but  the  task  of  figuring  out  the 
overall  capacity  for  leadership  performance  is  fundamentally  different  than  measuring  someone’s 
current  performance.  It  is  possible  that  over  time  correlations  would  be  begin  to  occur  such  that  if  a 
junior  officer  obtains  a  certain  score  then  their  overall  capacity  might  be  inferred.  However,  this  hne 
of  research  is  out  of  scope  for  this  proposal. 

The  main  objective  of  defining  the  assumptions  and  investigating  the  literature  on  leadership, 
assessment  centers  and  other  related  topics  was  to  determine  if  one  could  create  a  set  of 
measurement  dimensions  that  could  be  used  in  multiple  domains,  multiple  situations  and  with 
multiple  levels  of  user  experience.  Additional  research  was  conducted  to  create  a  theoretical 
framework  for  how  feedback  interventions  should  be  handled  with  the  overall  stated  goal  of 
increased  leadership  performance. 

4.1  Role  of  Feedback  on  User  Performance 

One  of  the  primary  issues  with  creating  a  computer-based  assessment  center  is  what  one  does 
with  the  data  that  is  collected  from  the  simulation  or  user  interaction  with  the  system.  Computer 
Based  assessment  centers  are  used  in  a  variety  of  locations  and  for  a  variety  of  reasons.  With  respect 
to  the  current  research.  Progeny  systems  had  to  determine  if  the  provided  solution  should  just  assess 
and  report  on  a  user’s  performance  or  if  the  system  should  also  have  some  functionality  to  help 
improve  users’  performance  over  mirltiple  trials  of  the  system. 


Number  of 
People 


Performance  on  the  Leadership  Construct 
Figure  6;  Leadership  Performance  Curve 
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The  literature  on  the  utility  of  feedback  interventions  for  increasing  performance  was  a  valuable 
line  of  research.  Of  particular  value  was  The  Effkts  cfFeedbcuk  IrOerwTtkm  m  Perfarrmnoe  (Kluger  and 
DeNisi  1996).  Kluger  and  DeNisi  discussed  how  certain  types  of  feedback  interventions  were 
beneficial  in  certain  arenas  and  could  in  fact  be  detrimental  to  performance  in  different  areas.  For 
example,  Kluger  and  DeNisi’s  research  shows  that  when  a  user  received  a  feedback  intervention,  his 
or  her  reaction  to  that  intervention  depends  on  where  on  the  standard  they  were  rated.  If  the  user 
was  rated  below  a  standard,  most  people  tended  to  increase  their  effort  to  get  up  to  the  standard 
measure.  However,  in  cases  where  the  person  scored  above  a  certain  measure  people  tended  to 
either  maintain  or  reduce  their  effort.  This  finding  applies  to  the  multi-dimensional  leadership 
construct  because  if  users  receive  feedback  that  they  meet  or  exceed  the  standard  on  certain 
dimensions  of  leadership  they  may  reduce  their  effort  on  those  dimensions  in  order  to  reallocate 
their  focus  to  dimensions  where  they  might  have  missed  the  standard.  Why  on  the  surface,  this 
approach  seems  logical,  the  users  may  actually  decrease  their  performance  because  as  discussed 
above,  not  aU  dimensions  are  weighted  equally,  the  interactions  be  dimensions  might  be  complex, 
some  dimensions  might  be  fixed  (i.e.  people  cannot  work  at  increasing  their  I.C^,  or  they  may 
“overshoot”  the  effort  necessary  and  thusly  not  only  continue  to  miss  the  standard  on  the  original 
dimension,  but  also  miss  the  standard  on  a  dimension  they  previously  “passed.” 

Lastly,  Kluger  and  DeNisi  discussed  the  impact  of  feedback  intervention  on  the  “locus  of 
attention”  for  users.  They  stated  that  if  a  feedback  intervention  shifted  the  focus  of  attention  from 
its  nominal  location  in  the  middle  of  the  Feedback- loop  Hierarchy  up  to  the  “Self”  component  then 
performance  is  generally  degraded.  They  posited  several  hypotheses  for  why  performance  degrades, 
but  with  respect  to  Progeny  proposed  solution,  it  is  sufficient  to  note  that  the  feedback  intervention 
should  try  to  focus  the  attention  down  the  hierarchy  to  the  “task  detail”  level.  Kluger  and  DeNisi 
also  made  suggestions  about  how  the  feedback  intervention  should  be  stmctured  so  that  that  the 
users  do  not  begin  to  modify  successful  decision  making  strategies.  Therefore,  it  is  imperative  that 
the  feedback  interventions  be  structured  such  that  the  users  receive  guidance  on  where  the  errors 
could  be  occurring  or  methods  that  might  lead  to  error  reduction. 

The  line  of  research  into  the  effects  of  feedback  intervention  have  specific  impacts  to  Progeny’s 
proposed  solution: 

•  Feedback  interventions  need  to  be  stmctured  such  that  it  enables  users  to  accurately 
determine  areas  where  additional  effort  is  most  likely  to  increase  performance  on  the 
leadership  constmct. 

•  Feedback  interventions  need  to  focus  user  on  tasks  and  provide  areas  of  improvement 

•  Feedback  interventions  need  to  change  style  and  format  depending  on  the  task  characteristic 

•  The  Leadership  constmct  is  a  “long  duration”  attribute  and  as  such  should  be  measured  over 
a  long  period  of  time 

o  Corollary;  When  “long  duration”  attributes  are  sufficiently  practiced,  existing  user 
resources  (i.e.  Intelligence,  Multi-tasking  constmct)  become  less  predictive. 

•  The  effect  of  Feedback  interventions  change  with  differences  in  situational  variables 

•  Personality  traits  of  the  users  change  the  impact  of  Feedback  interventions 
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4.2  Leadership  Behavior  Taxonomy 

The  literature  is  littered  with  different  sorts  of  leadership  behavior  taxonomies.  Some  of  the 
taxonomies  focus  on  the  general  effect  of  leaders.  For  example,  Fleishmen  (1953)h  Halpin  &  Winer 
(1957)2;  and  Blake  &  Mouton  (1982)^  categories  leadership  behavior  in  to  two  broad 
“metacategories.”  These  “metacategories”  can  be  best  described  as  relations- oriented  behavior  and 
task-oriented  behaviors.  Relations  oriented  behaviors  are  behaviors  that  influence  a  subordinates 
actions  based  upon  social  and  organizational  relationships.  For  example,  a  relation-oriented  style 
might  entail  a  manager  using  his  or  her  personal  relationships  with  a  subordinate  to  get  that 
subordinate  to  work  harder,  or  to  change  maladaptive  behaviors.  Task-oriented  behaviors  focus  less 
on  the  interpersonal  relationships  between  the  leader  and  the  subordinate,  and  focus  more  on  the 
specific  nature  of  the  task.  Task-oriented  behaviors  rely  on  well  defined  oi^anizational  structure,  a 
clear  hierarchy  or  chain  of  command,  and  well  defined  tasks  and  goals.  Yukl,  Gordon  and  Taber 
(2002)“’  argue  that  in  addition  to  task  and  relation  oriented  behaviors  a  third  metacategoiy  exists. 
This  third  category  is  best  described  as  change-oriented  behaviors.  Unlike  the  other  two 
metacategories,  change- oriented  behaviors  capture  the  more  coaching  or  performance  enhancing 
behaviors  leaders’  exhibit.  An  example  of  change  behavior  leadership  would  be  when  a  leader 
discusses  a  change  in  presentation  styde  to  a  subordinate,  or  more  topically,  when  a  commander 
suggests  a  change  in  tactics  or  actions  of  a  Soldier. 

Yukl,  Gordon  and  Taber  (2002)  define  12  specific  leadership  behaviors.  While  these  behaviors 
are  more  suited  for  civilian  leadership  than  military  leadership,  they  do  provide  a  framework  by 
which  Progeny  created  a  theoretical  framework: 

•  Clarifying  Roles:  Assigning  tasks  and  explaining  job  responsibilities,  objectives  and 
performance  expectations. 

•  Monitoring  operations:  Checking  on  the  progress  and  quality  of  the  work,  evaluation  of 
individual  and  unit  performance. 

•  Short-term  Planning:  Determining  how  to  use  personnel  and  resources  to  accomplish  a  task 
efficiently,  and  determining  how  to  schedule  and  coordinate  unit  activities  efficiently. 

•  Consulting:  Checking  with  people  before  making  decisions  that  affect  them.  Encouraging 
participation  in  decision  making  and  using  ideas  and  suggestions  of  others. 

•  Supporting:  Acting  considerate,  showing  sympathy  and  support  when  someone  is  upset  or 
anxious  and  providing  encouragement  and  support  when  there  is  a  difficult,  stressful  task. 

•  Recognizing:  Providing  praise  and  recognition  for  effective  performance,  significant 
achievements  and  special  contributions  and  performance  improvements. 

•  Developing:  Providing  coaching  and  advice,  providing  opportunities  for  skill  development, 
and  helping  people  learn  how  to  improve  their  skills. 

•  Empowering:  Allowing  substantial  responsibility  and  discretion  in  work  activities  and  trust 
people  to  solve  problems  and  make  decisions  without  getting  prior  approval. 


'  Fleishman,  E.  A  (1953).  The  description  of  supervisory  behavior.  Personnel  Psychology,  36, 1-6 

^  Halpin,  A.  W.,  Winer,  B.  J.  (1957).  A  factorial  study  of  the  leader  beha\rior  descriptions.  In  R.  M.  Stogdill  &  A.  E.  Coons  (EDs.),  Leader 
behavior:  Its  descriptions  and  measurement.  Columbus,  Oh:  Bureau  of  Business  Research,  Ohio  State  University. 

^  Blake,  R.R.,  &  Mouton,  J.  S.  (1982).  Management  by  grid  principles  or  situationalism:  Which?  Group  and  Organization  Studies,  7, 207-210 
Yukl,  G.,  Gordon,  A.,  &  Taber,  T.  (2002).  A  Hierarchical  Taxonomy  of  Leadership  Behavior:  Integrating  a  Half  Cnetury  of  Behavior 
Research.  Journal  of  Leadership  and  Organizational  Studies,  Vol.  9, 1, 15-32. 
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•  Envisioning  Qiange:  Presenting  an  appealing  description  of  desirable  outcomes  that  can  be 
achieved  by  the  unit,  describing  a  proposed  change  with  great  enthusiasm  and  conviction. 

•  Taking  Risks  For  Qiange:  Taking  personal  risks  and  making  sacrifices  to  encourage  and 
promote  desirable  change  in  the  organization. 

•  Encouraging  Innovative  Thinking:  Qiallenging  people  to  question  their  assumptions  about 
the  work  and  consider  better  ways  to  do  it. 

•  External  Monitoring:  Analyzing  information  about  events,  trends  and  changes  in  the 
external  environment  to  identify  threats  and  opportunities  for  the  organization  unit. 

Yukl,  Gordon  and  Taber  (page  25, 2002) 

4.3  Naturalistic  Decision  Making 

The  previous  two  bodies  of  literature  were  focused  on  how  to  provide  feedback  to  users,  and  a 
potential  framework  for  leadership  behaviors  that  could  be  captured.  This  section  of  literature  was 
based  upon  how  one  would  create  a  scenario  to  assess  users  on  the  leadership  constmct.  Progeny 
Systems  had  the  goal  of  creating  scenarios  for  different  environments  in  a  quick  and  efficient 
manner.  One  of  the  major  limitations  of  many  existing  simulation  or  computer-based  environments 
is  that  it  takes  an  inordinate  amount  of  time  to  develop  new  material  when  enemies  change  their 
tactics  or  when  new  combat  theatres  emerge.  Progeny’s  goal  was  to  create  situations  that  transcend 
situational  details  such  that  if  a  combat  theatre  was  in  a  jungle  or  a  desert,  only  small,  unobtmsive 
aspects  of  the  tool  need  to  change.  The  concept  was  that  while  tactics  might  change  dramatically 
based  on  terrain,  or  environmental  cues,  the  act  of  leadership  would  more  or  less  stay  unchanged. 
In  an  effort  to  define  the  characteristic  of  a  situation  that  are  essentially  immune  to  external 
situational  characteristics.  Progeny  investigated  the  literature  surrounding  naturalistic  decision 
making.  Klein  et.  al.^  proposed  10  traits  or  features  that  lead  to  naturalistic  decision  making: 

•  Ill-defined  goals  and  ill-structure  tasks 

•  Uncertainty,  ambiguity  and  missing  data 

•  Shifting  and  competing  goals 

•  Dynamic  and  continually  changing  conditions 

•  Action- feedback  loops  (real-time  reactions  to  changed  condition) 

•  Time  stress 

•  High  Stakes 

•  Multiple  players 

•  Organizational  goals  and  norms 

•  Experience  decision  makers 

The  Klein  et.  al.  model  fits  will  within  the  military  domain  as  none  of  these  characteristics  are 
violated.  Through  the  use  of  these  features,  Progeny’s  proposed  solution  allows  for  a  leadership 
assessment  of  an  individual  independent  of  situational  variables  such  as  operational  theatre,  enemy 
tactics  or  other  similar  characteristic.  Via  variable  independence.  Progeny’s  proposed  solution 


®  Klein,  G.  A.,  Orasanu,  J.,  &  Calderwood,  R.  (1993)  Decision  Making  in  action:  Modeis  and  methods.  Norwood,  NJ:  Ablex  Publishing 
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provides  a  greater  ability  to  create  simulations  or  assessment  scenarios  quickly  and  in  a  wide  domain 
while  keeping  controlling  internal  validity. 

5  Conclusions 

The  research  developed  as  part  of  this  Phase  I  effort  highlighted  that  while  the  literature  has 
numerous  theories  on  leadership  and  taxonomies  for  categorizing  leadership  behaviors,  there  is  very 
little  work  on  actually  measuring  and  creating  a  predictive  capability  for  assessing  a  person’s 
leadership  potential.  Progeny  Systems’  approach  was  to  use  gaming,  web,  and  learning 
management  technologies  to  produce  a  proof-of-concept  assessment  center  for  the  multi¬ 
dimensional  leadership  constmct.  The  following  section  describes  the  results  of  the  Phase  I  effort: 

While  the  Leadership  constmct  is  multi-faceted,  its  different  dimensions  can  be  measured  effectively 
via  an  assessment  center  whose  foundation  is  a  commercial  learning  management  systems  (LMS) 

Current  Learning  management  systems  are  able  to  present  users  with  a  variety  of  multi-modal 
learning  and  testing  options.  Through  the  use  of  commercial  LMSs  Progeny  has  the  ability  to  take 
existing,  mature  paper  or  computer  based  assessment  batteries  for  any  of  the  dimensions  in  the 
leadership  constmct  and  both  present  them  to  users  and  store  the  results  for  assessing  the  different 
dimensions  of  leadership. 

Through  an  innovative  adaptation  of  the  SCORM  2004  standard.  Progeny  also  worked  through 
the  concept  of  delivering  simulation  and  other  assessment  stimulus  in  a  “chunked”  presentation 
format.  By  “chunking”  the  data.  Progeny  Systems  plans  to  be  able  to  create  a  more  granular, 
prescriptive  presentation  capability  than  what  currently  in  the  industry.  To  state  differently,  instead 
of  having  a  large  simulation  and  assessment  batteries  that  tests  all  users  in  the  same  way.  Progeny 
proposes  to  present  different  aspects  of  simulation  and  assessment  batteries  based  upon  user 
experience,  training  and  other  characteristics  that  vary  from  person  to  person. 

By  distinguishing  between  situational  variables  that  only  provide  context  to  decision  making  from 
those  variables  that  actually  can  be  used  to  assess  different  dimensions  of  the  leadership  constmct 
one  can  reach  a  high  degree  of  reuse 

The  literature  has  defined  characteristic  of  situations  that  can  be  used  to  smdy  a  person’s 
decision  making  ability.  Therefore  one  can  constmct  situations  where  one  could  measure  specific 
aptimdes  or  problem  areas  in  a  user  while  also  retaining  the  ability  to  vary  extraneous  simational 
characteristics.  In  other  words,  one  could  develop  a  situation  where  the  fundamental  measure  of  a 
leadership  dimension  is  unchanged,  but  external  variables  like  weather,  location,  and  enemy  tactics 
could  all  be  changed.  This  methodology  allows  for  a  high  degree  of  material  reuse  since  one  training 
or  simulation  environment  could  be  changed  so  as  to  almost  be  unrecognizable  from  the  original 
training  environment,  but  still  measure  the  same  dimension  of  interest.  This  innovation  provides  a 
huge  cost  savings  as  the  sponsor  could  reuse  material  in  a  jungle  theatre  that  was  originally 
developed  for  a  Desert  or  Artie  environment. 

Both  immersive  and  texmal  “simulation”  can  be  used  to  measure  dimensions  of  the  leadership 
constmct 

The  gaming  industry  as  well  as  a  good  amount  of  the  academic  research  has  focused  on 
immersive  simulations  in  order  to  train  users  in  certain  behaviors  or  to  smdy  and  assess  how  a  user 
will  react  when  given  a  certain  set  of  stimulus.  ''JClth  respect  to  the  leadership  constmct,  one  can 
gain  information  about  a  user  along  the  whole  continuum  from  highly  immersive,  to  texmal 
representations  of  problem  spaces.  Progeny  proposes  to  start  with  a  texmal  interaction  with  the 
user  to  capmre  decision  making  and  relevant  behaviors  and  evenmally  to  mature  into  a  more 
immersive  technology  realm.  Textual  representation  facilitates  quick  scenario  development  as  well 
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as  a  more  controlled  solution  set  that  affords  vastly  reduced  complexity  during  user  assessment.  As 
technologies  mature  and  Progeny’s  understanding  of  the  domain  increases,  the  ability  to  gather 
information  in  real-time  from  a  user’s  “behavior”  in  a  immersive  simulation  will  increase  the 
external  validity  of  the  assessment  as  well  as  provide  a  more  granular  assessment  due  to  the 
exponentially  increased  size  of  the  potential  solution  set.  In  other  words,  the  textual  representation 
provides  a  only  a  handful  of  behavioral  options  while  an  immersive  simulation  provides  nearly 
unlimited  options. 

6  Phase  II  Technical  Objectives  and  Approach 

The  primary  objective  of  the  proposed  Phase  II  effort  is  to  develop  a  prototype  leadership 
assessment  center.  In  order  to  create  an  assessment  center  of  any  type  it  is  necessary  to  define  the 
dimensions  by  which  one  is  going  to  assess  users.  In  the  specific  case  of  the  multi- dimension 
constmct  of  leadership,  one  needs  to  create  a  multi  dimensional  assessment  center  that  is  able  to 
measure  user  performance  on  individual  dimensions  while  also  collecting  the  results  from  the  entire 
dimension  to  come  up  with  an  integrated  “raw”  assessment  of  the  overall  constmct.  The  proposed 
innovation  will  utilize  a  methodology  similar  to  that  of  the  Navy’s  Five  Vector  Model  that  tracks 
individual  Sailor  progression  along  five  distinct  dimensions  in  order  to  determine  the  individual’s 
relative  performance  to  his  or  her  peers.  The  assessment  center  will  provide  a  raw  “score”  for  each 
identified  dimension  that  then  rolls  up  for  the  overall  assessment.  Depending  on  the  needs  of  the 
Sponsor  the  results  of  the  individual  dimensions  could  be  shared  with  the  user  as  part  of  a  feedback 
intervention,  or  could  be  kept  in  confidence  so  as  to  not  allow  a  user  to  “game”  the  system  or 
potentially  reduce  the  users  performance  consistent  with  the  research  described  in  section  1.4.1. 

An  important  task  that  is  related  to  the  primary  objective  is  the  definition  of  the  different 
dimensions  of  leadership.  A  validated  set  of  dimensions  might  even  be  considered  the  starting  point 
of  a  fully  implemented  assessment  center.  Progeny’s  approach  is  to  provide  the  sponsor  with  the 
capability  to  easily  create,  maintain  and  track  dimensions  as  research  continues.  Through  the  use  of 
a  Service  Oriented  Architecture  (SOA)  and  a  commercial  Learning  Management  System  (LMS), 
Progeny  already  has  all  the  tools  necessary  for  fleshing  out  the  assessment  center  with  as  many 
dimensions  as  the  Sponsor  deems  necessary.  So  instead  of  proposing  to  define  the  list  of  all  the 
dimensions  of  leadership.  Progeny  is  proposing  innovation  in  defining  and  capturing  behaviors  that 
are  predictive  for  leadership,  providing  feedback  interventions  that  are  tailored  to  the  situation  and 
user  and  creating  a  method  for  rapid  deployment  of  scenarios  by  creating  simulations  or  scenarios 
that  measure  leadership  behaviors  in  a  situational- agnostic  manner.  It  is  argued  that  through  the  use 
of  existing  Progeny  technologies,  systems  and  experience,  additional  dimensions  can  be  added  easily. 
For  example,  a  probable  dimension  for  predicting  leadership  performance  is  general  aptitude  (i.e. 
intelligence).  ITiere  are  numerous  IQ  tests  that  are  web  based,  (IEEE  standards  Compliant)  that 
could  “play”  on  Progeny’s  existing  infrastmcture.  If  general  aptitude  is  identified,  then  it  is  a  simple 
issue  to  integrate  that  existing  application  into  the  proposed  solution.  Another  example  might  be  a 
dimension  for  Leader  empathy.  It  is  hard  to  define  a  computer  based  method  for  measuring  a  user’s 
empathy.  Despite  the  difficulty  in  quantifying  such  a  concept,  there  are  numerous  paper  based  and 
interview- style  instruments  that  try  to  provide  a  quantified  score  of  empathy.  If  the  sponsor 
chooses  a  paper  based  instrument,  then  it  is  a  simple  step  to  convert  the  instrument  to  a  web  based 
tool.  It  could  also  be  possible  through  a  more  immersive  simulation  environment  to  track  exactly 
what  a  user  does  within  the  scenario.  Some  of  the  gaming  engines  described  the  previous  section 
have  a  first  person  perspective  so  it  would  be  possible  to  capture  if  a  user  “looks”  at  subordinates 
“talks”  with  them  or  just  “commands”  them  into  action.  Progeny  proposes  to  support  even 
instmctor  led,  free-form  interview  style  assessment  instruments  via  an  integration  stroke  with  a 
commercial-based  LMS.  Progeny  proposes  to  spend  the  effort  on  the  creation  of  the  infrastmcture 
necessary  for  a  fully  functional  assessment  center  that  could  then  be  used  for  other  constmcts 
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besides  leadership,  instead  of  conducting  the  research,  creating  Critical  incidents  and  holding  focus 
groups  necessary  to  create  another  description  of  the  leadership  construct.  That  being  said  Progeny 
Systems  intends  to  further  review  the  leadership  literature  as  well  as  work  with  the  sponsor  and 
other  subject  mater  experts  to  develop  and  define  a  subset  of  the  overall  leadership  dimensions. 

The  purpose  of  having  some  of  the  dimensions,  but  not  an  all  inclusive  set  is  two  fold.  First,  it 
allows  Progeny  to  accurately  test  multi-modal  (i.e.  Web-based  content,  translating  paper  based 
content,  interview- style  content,  etc.)  dimensions.  Through  the  prototyping  of  multi-modal 
dimensions  Progeny  will  demonstrate  the  capabilities  of  its  existing  technologies  and  systems  and 
measure  their  applicability  to  the  general  concept  of  assessment  centers.  The  second  benefit  is  that 
the  prototype  provided  to  the  sponsor  will  have  some  predictive  ability  and  potentially  could  be 
used  with  real  users  while  additional  dimensions  are  defined  and  validated.  Essentially,  the  approach 
provides  the  sponsor  with  a  phased  implementation  plan  that  allows  for  refining  the  proposed 
solution  at  multiple  points  and  as  more  data  becomes  available. 

6.1  Embodiment  of  Theoretical  Foundations 

Progeny  has  developed  a  functioning  proof  of  concept  of  the  system  described  in  section  1.1 
that  is  based  on  the  research  laid  out  in  the  section  1.4.1,  1.4.2  and  1.4.3.  The  proof  of  concept 
application  shows  that  Progeny  can  take  a  scenario  as  described  in  the  66  Stones  cf  Battle  Command 
and  capture  user  inputs,  report  out  the  inputs  and  score  them  against  a  predefined  construct  of 
leadership.  The  follow  sections  describe  how  Progeny  proposes  to  mature  those  capabilities  in  a 
Phase  II  environment 

6.1.1  Leadership  Behaviors 

Progeny  proposes  to  continue  to  refine  the  concepts  behind  the  leadership  construct  by  more 
fully  fleshing  out  the  list  of  behaviors  that  typify  effective  or  highly  effective  leadership.  It  is  out  of 
scope  to  create  a  fully  qualified  (or  validated)  list  of  leadership  behaviors,  but  a  representative  subset 
could  be  defined  through  a  number  of  potential  methods.  One  method  would  be  to  conduct  a 
focus  group  and  follow  the  standard  critical  incident  technique.  However,  such  a  method  would 
require  many  subject  matter  experts  and  well  as  several  hundred  critical  incidents  in  order  to  meet 
the  statistical  requirements.  Another  method  would  be  to  utilize  the  dimensions  described  in  the 
FM22-100  as  a  foundation  for  the  creation  of  a  finite  set  of  behaviors.  This  method  would  utilize 
the  dimensions  that  the  Army  has  identified  as  key  to  the  leadership  construct  and  provide  a 
facilitated  mapping  between  leadership  dimension  and  measured  behavior.  Some  effort  would  have 
to  be  expended  to  verify  and/ or  validate  that  the  behaviors  identified  for  each  dimension  actually 
measured  that  dimension,  but  that  effort  should  be  nominal.  One  last  approach  would  be  to  further 
investigate  the  literature  and  try  to  determine  which  behaviors  have  been  studied  in  the  past  and 
how  those  behaviors  map  to  the  dimensions  listed  in  FM22-100.  Progeny  proposes  to  utilize  a 
combination  of  the  last  two  methods  described.  The  research  conducted  as  part  of  the  Phase  I 
effort,  showed  that  one  could  create  a  behavior  taxonomy  by  investigating  the  literature,  but  that  the 
literature  was  not  always  well  suited  for  military  domains.  Issues  like  strict  hierarchy  and  compelled 
compliance  are  not  prevalent  in  the  civilian  world  or  the  research  that  studies  civilian  leadership 
behaviors.  The  proposed  methodology  for  creating  the  measurement  behaviors  for  the  leadership 
construct  is  sufficient  for  Progeny’s  overall  objective  of  creating  a  proto-type  leadership  assessment 
center.  For  a  fully  mature  assessment  center,  one  would  have  to  be  assured  that  all  behaviors  were 
captured  and  assessed.  However,  with  the  bar  somewhat  lowered  by  prototyping  the  design 
Progeny  is  concerned  more  with  validating  the  approach,  rather  that  having  a  100%  valid 
assessment.  That  being  said,  it  is  hoped  that  even  though  the  proposed  solution  is  less  than  fully 
validated,  it  would  still  offer  some  predictive  or  diagnostic  capabilities. 

6.1.2  Feedback  Intervention 
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The  literature  describe  above  gives  a  fairly  detailed  accounting  and  framework  for  how 
feedback  should  be  given  to  user  depending  on  the  situation  faced  by  the  user  and  depending  on  the 
overall  task  characteristics.  In  order  to  maximize  user  performance,  the  types  of  feedback,  the 
interval  of  when  to  provide  feedback  and  the  level  of  specificity  of  the  feedback  all  need  to  be 
variable.  Progeny  proposes  to  create  a  codified  methodology  that  take  user-specific  attributes  (i.e. 
training,  experience,  previously  defined  dimensions  of  the  leadership  construct)  and  situation 
specific  attributes  to  define  the  relevant  feedback  intervention  characteristics.  In  other  words,  the 
proof- of- concept  already  is  able  to  change  complexity  of  the  scenario  based  upon  user-specific  data, 
so  it  is  a  logical  extension  that  the  types  of  feedback  could  also  change  based  upon  this  data.  For 
example,  the  literature  described  how  the  more  practiced  a  task  was  the  less  of  an  impact  a  feedback 
intervention  had  in  certain  situations.  In  those  situations  a  junior  officer  might  have  a  feedback 
intervention  while  a  less  junior  officer  received  no  feedback  at  all.  The  feedback  intervention 
strategy  would  tiy  to  optimize  performance  by  avoiding  known  problems  areas.  To  date,  variable 
feedback  intervention  is  not  prevalent  in  the  leadership  domain  and  represents  an  innovation  that 
could  provide  the  proposed  solution  a  marked  improvement  over  other  solutions  that  only  provide 
one  type  of  feedback  or  that  do  not  utilize  all  available  and  relevant  data  about  the  user.  This 
concept  could  be  further  extended  to  use  personality  traits,  educational  background,  or  even 
learning  styles  for  how  feedback  intervention  is  handled. 

6.1.3  Naturalistic  Decision  Making 

The  proof  of  concept  developed  for  Phase  I  utilized  the  66  Stones  cf  Battle  Qmmndio  develop  a 
meaningful  scenario  to  test  out  the  concepts  developed  within  the  overall  research  line.  However, 
for  the  more  robust  system  to  be  done  as  part  of  the  Phase  II  effort.  Progeny  proposes  to  utilize  the 
approach  described  in  the  naturalistic  decision  making  literature  to  create  situations  that  transcend 
situational  characteristics.  In  other  words,  tactics  might  change  dramatically  based  upon  something 
like  terrain,  but  how  a  leader  handles  a  scared  subordinate  or  exhibits  other  leadership  behaviors 
arguably  do  not  change  based  upon  such  variables.  Progeny  proposes  to  use  the  ten  features  of 
naturalistic  decision  making  as  constraints  for  scenario  development.  In  other  words,  a  scenario 
could  be  developed  like  one  in  the  proof  of  concept  or  like  one  of  the  stories  in  the  66  Stories  cf  Battle 
Command  that  uses  features  like  time  stress,  ill-structured  tasks,  and  high  stakes  to  elicit  assessable 
leadership  behaviors.  However,  unlike  the  existing  stories  which  provide  the  details  for  how  a 
person  could  be  assessed.  Progeny  proposes  that  situational  details  just  become  the  context  of  the 
assessment.  Like  the  proof  of  concept,  the  proposed  prototype  would  measure  the  presence  or 
absence  of  defined  behaviors,  and  methods  of  leadership  exhibited  by  the  user.  The  description  of 
the  weather,  location  or  other  variable  would  only  be  value  as  a  framing  concept  to  the  user.  This 
methodology  would  allow  for  very  rapid  scenario  creation  as  the  collection  of  situations  could  be 
stored  and  then  replayed  in  different  environments  or  theatres  as  enemies  change  their  tactics  or  as 
new  threats  emerge.  This  methodology  stands  in  stark  contrast  to  solutions  that  require  long  lead 
times  due  to  the  highly  coupled  nature  of  the  situational  variables  to  the  assessment  instrument. 

7  Phase  II  Work  Plan 

7.1  Define  subset  of  Leadership  Dimensions 

In  order  to  provide  the  sponsor  with  a  functioning  and  useful  prototype.  Progeny  must  define  a 
set  of  dimensions  that  could  be  used  in  the  assessment  of  the  leadership  construct.  In  order  to 
provide  a  better  demonstration  of  the  capabilities  of  the  proposed  innovation,  the  defined 
dimensions  will  be  multi-modal  (i.e.  Web  content  and  paper-based  content).  In  order  to  define  the 
selected  dimensions.  Progeny  will  utilize  a  thorough  literature  review,  work  with  subject  matter 
experts  as  well  as  work  with  the  sponsor  to  determine  if  certain  dimensions  are  more  interesting  or 
Hne  up  with  related  Army  research  topics. 
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7.2  Integrate  Leadership  Dimensions  with  existing  Progeny 
Technologies 

Once  the  dimensions  are  defined,  they  must  be  integrated  into  the  overall  solution.  It  is 
assumed  that  this  effort  will  be  trivial,  but  necessary  in  order  to  prove  out  the  overall  concepts  of  the 
proposed  solution  set 

7.3  Defined  Leadership  Behaviors 

As  with  leadership  dimensions  but  on  a  grander  scale,  there  are  many  more  leadership 
behaviors  that  could  be  captured  and  measured  than  are  described  in  the  previous  sections.  Progeny 
has  identified  twelve  high-level  behaviors  based  upon  a  literature  review.  These  12  behaviors  need 
to  be  further  refined  such  that  each  of  the  high  level  behaviors  has  3  or  4  sub  behaviors.  By 
increasing  the  number  of  behaviors.  Progeny’s  solution  also  for  extra  granularity  in  the  assessment 
of  how  an  individual  exhibits  leadership.  Additionally,  if  the  future  research  shows  that  one  or  more 
of  the  sub-behaviors  are  invalid  measures,  there  are  still  items  mapped  to  each  high  level  behavior. 

7.4  Implement  Variable  Feedback  Technique 

As  described  above,  one  of  the  proposed  innovations  of  Progeny’s  solution  is  the  ability  to 
change  the  type,  style  and  temporal  location  of  a  feedback  innovation.  The  relevant  characteristics 
of  the  user  and  the  situation  need  to  be  captured  and  made  available  to  the  system.  Related  to  this 
item  are  tasks  associated  with  the  creation  of  a  taxonomy  of  complexity  for  a  particular  scenario. 
Kluger  and  DeNisi  provide  one  methodology  to  assigning  a  complexity  level  to  a  situation.  Progeny 
systems  intends  to  research  additional  methodologies  and  pick  the  “best  of  breed”  so  to  quickly, 
accurately  and  consistently  assign  a  complexity  rating  to  a  given  scenario. 

7.5  Naturalistic  Scenarios 

The  proof  of  concept  described  in  Phase  I  result  section  shows  a  tool  for  creating  scenarios  in  a 
relatively  easy  and  straight  forward  manner.  For  the  prototype  solution,  this  scenario  engine  will  be 
extended  to  encompass  the  concepts  described  in  Naturalistic  Decisions  Making  in  order  to  create 
assessment  scenarios  that  are  not  tightly  coupled  to  situational  variables.  The  ten  traits  of  naturalistic 
decision  making  will  be  incorporated  into  the  proposed  innovation  such  that  a  scenario  developer 
can  create  an  assessment  event  for  one  situation  and  then  reuse  the  same  event  for  a  different 
situation  at  a  later  date. 

7.6  Evolve  Simulation  Deiivery  System 

Progeny  Systems  will  continue  the  effort  started  with  the  Phase  I  Prototype  Simulation  Delivery 
System  to  provide  an  easily  deployable  and  effective  Leadership  Training  Simulation  Environment. 
The  development  process  will  target  three  main  sections. 

7.6.1  Immersive  Simulation  Engine 

During  the  Phase  I  effort  Progeny  Systems  developed  a  Prototype  Simulation  Engine  to 
demonstrate  the  back-end  data  access  web  services.  Progeny  will  work  with  Commercial  Vendors  to 
either  extend  the  capabilities  of  the  Prototype  Simulation  Engine  or  completely  replace  it.  The  end 
goal  is  to  provide  Simulation  Authors  an  immersive  Simulation  Environment  that  is  capable  of 
measuring  more  Leadership  Characteristics  than  the  original  Prototype. 

7.6.2  Simulation  Script  Capabilities 

Progeny  will  evaluate  the  feasibility  of  utilizing  the  Prototype  Simulation  Script  format  with  the 
new  Immersive  Simulation  Engine.  Based  on  the  feasibility  evaluation  results  Progeny  will  either 
extend  the  Commercial  Vendor’s  Simulation  Authoring  format  to  utilize  Simulation  Adjustment 
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Parameters  or  extend  the  existing  Simulation  Script  format  to  support  the  advanced  Leadership 
Qiaracteristic  measurement  capabilities  found  in  the  new  Immersive  Simulation  Engine. 

7.6.3  Leadership  Performance  Measurements 

One  of  the  primaiy  thrusts  of  the  Phase  I  effort  involved  researching  Leadership  Performance 
measurement  techniques.  While  Progeny  determined  an  effective  method  for  assessing  Leadership 
skills  in  a  Simulation  Environment,  the  selected  metrics  were  limited  by  the  Simulation  Engine 
technology.  For  the  Phase  II  effort  Progeny  will  revisit  the  Phase  I  research  to  identify  new  metrics 
that  are  available  due  to  the  new  Simulation  Engine  technology.  Examples  of  potential  new 
measurable  Leadership  behaviors  are  effective  Eye  Contact  and  Listening  ability. 

7.7  Integrate  Leadership  Assessment  Center  with  AKO 

One  of  the  main  goals  of  the  entire  A05-025  project  is  to  easily  deploy  Leadership  SkiQs 
Assessments  and  Simulations  to  the  Soldiers.  Rather  than  spend  time  and  money  developing  a 
proprietary  delivery  mechanism  Progeny  proposes  to  utilize  the  Army's  existing  learning  content 
distribution  tool.  Army  Knowledge  Online  provides  a  standardized  way  for  Soldiers  to  access 
learning  content.  Progeny  will  utilize  their  experience  with  Navy  Knowledge  Online  to  integrate  the 
Prototype  Leadership  Assessment  Center  with  the  AKO  LMS. 

7.7.1  Research  the  Army  LMS 

Progeny  will  begin  this  effort  by  thoroughly  researching  the  LMS  capabilities  provided  within 
AKO.  Progeny  will  utilize  existing  relationships  with  Commercial  Vendors  to  address  any 
shortcomings  in  the  AKO  LMS.  The  LMS  research  will  dictate  how  the  other  sections  of  this  effort 
are  conducted. 

7.7.2  Learner  Training  Records 

The  Phase  I  effort  required  Progeny  to  define  the  format  used  for  storing  records  of  Leadership 
Skills.  Progeny  elected  to  model  it  off  of  NKO  st^e  Learner  Skill  records.  Based  upon  the  Army 
LMS  research  the  Learner  Training  Record  format  may  need  to  be  modified  or  expanded  to 
function  in  the  Army  environment.  Progeny  will  work  to  mimic  AKO  style  performance 
measurement  formats  with  the  updated  Learner  Training  Records. 

7.7.3  Utiiize  SCORM  2004  for  Simulation  Delivery 

The  Phase  I  Simulation  Environment  relied  on  web  services  for  delivering  the  simulation 
content  to  the  web  based  Simulation  Engine.  For  the  Phase  II  effort  Progeny  intends  to  utilize  a 
standard  method  for  delivering  content  that  is  supported  by  most  LMS  and  LCMS  solutions. 
Progeny  will  research  the  feasibility  of  encapsulating  the  Simulation  Content  in  a  SCORM  2004 
wrapper.  Progeny  will  utilize  an  existing  relationship  with  Rustici  Software  to  aide  with  SCORM 
2004  standard  requirements.  Rustici  Software  has  worked  with  Progeny  in  the  past  to  provide 
SCORM  expertise.  Rustici  Software’s  SCORM  Content  Player  is  currently  utilized  by  the  Navy  and 
Marines  for  rendering  their  SCORM  courseware. 

7.7.4  Deploy  Prototype  Leadership  Assessment  Center 

Progeny  will,  with  the  help  of  the  TPOQ  establish  a  demonstration  environment  that 
showcases  the  Prototype  Leadership  Assessment  Center  integrated  with  AKO.  This  task  will 
require  considerable  coordination  between  Progeny  and  various  Army  officials.  At  a  minimum 
Progeny  will  deploy  the  Prototype  Leadership  Assessment  Center  using  the  applicable  subsets  of 
AKO  or  similar  competing  products.  The  intent  is  to  demonstrate  all  of  the  discussed  Leadership 
Assessment  Center  capabilities  in  an  easily  deployable  environment. 
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7.8  Schedule 


Program  Milestones 


•  Program  Go-Ahead  and  Final  Report 
•Status  Reports  and  Meetings 
•Demonstrations 


1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  18  17  18  19  20  21^22  23  24 


Pwsram  Tasks 


•Define  Leadership  Dimensions 
•Integrated  Dimension  Instruments 
•Define  Leadership  Behaviors 
•Naturalistic  Scenarios 
•Variable  Feed  Back  Techniques 
•Evolve  Simulation  Delivery  System 
•Integrate  Assessment  Center  with  AKO 
•Research  AKO  LMS 

•Utilize  SCORM  2004  For  Delivery  Package 
•Deploy  Prototype 
•Validate  and  Verify  Prototype 
•Capture  Future  Work  _ 


1 1 1 1 : 
1 1 1 1 
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8  Related  Work 

8.1  Non-Tactical  Data  Processing  System  Software  Integration 

Naval  Sea  System  Command  -  Contract  No.  N00024-98-G6226 

Progeny  is  responsible  for  the  integration;  testing  and  certification  of  the  NTDPS  software  for 
delivery  to  the  VIRGINIA  program  (Electric  Boat  for  Contractor  Furnished  System)  as  well  as  to 
the  back-fit  fleet  through  NUWC  Newport  and  PMW  165  for  SubLAN  installation  on  688  class 
submarines.  This  effort  pulls  together  numerous  Government  and  commercial  software 
applications  into  a  coordinated  product  with  functionality  defined  by  the  NTDPS  “A”  Specification 
that  is  agreed  to  by  TYCOMS/SEA08.  Progeny  is  responsible  for  configuration  management,  risk 
management,  and  delivery  of  the  various  software  baselines.  Tasking  includes  development  of  an 
end-to-end  (ship-to-shore)  solution  leveraging  the  VIRGINIA  class  Non-Tactical  Data  Processing 
System  Paperless  Ship  Applications,  Submarine  Non-Tactical  Application  Delivery  Interface  System 
(SNADIS),  and  TDKM  products  with  delivery  to  SPAWAR  PMW  165  and  NUWC  for  certification 
and  fielding  to  SSN  688  and  VIRGINIA  class  submarines  starting  in  Oct  2004.  The  effort  also 
includes  the  development  of  a  non-tactical  point  of  delivery  system  required  for  enhancing  the 
quality  of  work  on  board  navy  vessels  by  delivering  just-in-time  training  and  other  learning  products 
via  the  Integrated  Shipboard  Learning  Environment  (ISLE).  This  effort  is  being  done  in 
cooperation  with  the  Submarine  Learning  Center  (SLC)  (establishing  fimctional  requirements)  and 
uses  the  TDKM  product  for  shore-to-ship  content  synchronization.  Progeny  has  developed  the 
concept  of  operations  for  the  Submarine  Learning  Center  TDKM  wholesaler  and  Content 
Management  Server  and  is  working  on  the  issue  of  shore-to-ship  connectivity.  This  project  contains 
identical  functionality  as  being  constructed  on  the  shore-side  infrastructure  for  the  NAVSEA’s 
Distance  Support  Initiative  and  the  CNO’s  Revolution  in  Training. 
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8.2  Advanced  Logistics  Using  Multimedia  Technology 

Naval  Sea  System  Command  -  Contract  No.  N00024-03-G4104  &N00024-04-G4194 

Under  this  Phase  II  Small  Business  Innovative  Research  (SBIR)  contract,  Progeny  Systems  will 
provide  the  capabilities  of  the  Semantic  Web  to  TD/KM.  That  is,  provide  a  semi-structured 
information  infrastructure  by  adding  semantic  context  about  the  learning  and/ or  technical  content. 

This  effort  addresses  development  of  high-level  requirements  such  that  shipboard  content 
management  system  functionality  can  be  implemented  using  SCORM  and  SIOOOD  for  reuse  of 
technical/ learning  content.  This  effort  also  addresses  the  development  of  system/software 
architecture  for  the  contextualization,  discovery  and  resolution  of  learning/ technical  objects. 
Progeny  Systems  in  conjunction  with  the  current  TDKM  program  deriving  a  set  of  compatible 
metadata  technologies  that  can  be  used  for  information  metadata  and  information  semantics,  and 
their  associated  schema  and  predicate  languages.  This  project  also  entails  development  of  an 
Information  Object  Framework  that  permits  disparate  metadata  technologies  to  interoperate  thus 
enabling  platform  integration  across  providers  and  within  affordable  technology  refresh  cycles.  This 
includes  development  of  a  semi- automated  process  to  create  information  semantics  using 
information  metadata  to  accelerate  initial  development  and  to  provide  an  infrastructure  for 
collaboration. 

This  SBIR  also  includes  identifying  and  relating  the  most  relevant  technologies  and 
specifications  that  can  be  applied  to  TD/KM  and  Distance  Support,  in  line  with  the  developed 
architecture.  As  well  as  developing  a  set  of  prototype  components  that  can  demonstrate  selected 
functionality  from  an  end-to-end,  i.e.,  shore  based  content-to-performance  enhancement  in  the 
hands  of  the  Sailor  (Submarine  Knowledge  Management). 

8.3  Hybrid  Sailor  Performance  and  Training 

Progeny  proposes  to  create  a  set  of  constructs  necessary  for  the  assessment  and  selection  of 
Sailors  for  a  new  class  of  surface  combatant:  the  Littoral  Combat  Ship  (LCS).  The  LCS  has  a  stated 
concept  of  operations  to  have  a  reduced  crew  complement  while  simultaneously  having  a  larger  than 
average  mission  profile.  In  order  to  meet  this  goal,  the  Navy  awarded  Progeny  a  Phase  I  contract  to 
create  a  set  of  constructs  that  measure  a  Sailors  ability  to  do  multiple  roles  and  be  a  “hybrid”  of 
current  roles  and  rates.  In  other  words,  the  preferred  sailor  will  not  only  be  a  skilled  sonar  operator, 
but  also  have  aptitudes  for  troubleshooting,  and  have  a  high  tolerance  of  ambiguity.  The  complete 
list  of  constructs  is  stiU  underdevelopment,  but  once  completed.  Progeny  will  create  a  plan  to 
conduct  a  Job  Analysis  necessary  to  validate  its  models  and  develop  the  knowledge,  skills  and 
abilities  (KSA)  used  for  testing  “human  attributes”  that  are  not  currently  tested. 

One  construct  that  has  been  identified  is  that  of  “leadership.”  The  research  for  the  Army  SBIR 
and  for  this  Navy  SBIR  is  synergistic  and  provides  a  mechanism  for  cross  pollination  between  the 
two  branches  of  the  armed  services. 

9  Relationship  with  Future  Research  or  Research  and 
Development 

Progeny  Systems  has  several  efforts  that  are  related  to  the  assessment  of  individuals  on 
multidimensional  constmcts.  One  effort  is  with  the  Navy  and  entails  the  assessment  and  later  the 
selection  of  Sailors  who  will  perform  “hybrid”  tasking  on  the  new  surface  combatant  (the  Littoral 
Combat  Ship.)  Additionally  Progeny  is  working  with  different  Navy  and  Marine  Commands  to 
work  out  the  details  and  plans  necessary  for  utilizing  and  integrating  Progeny’s  Afloat  Integrated 
Learning  Environment  with  other  tools  and  software  like  Question  Mark 
(http:/ / www.questionmark.com/ us/home.htm)  to  provide  additional  assessment  capabilities. 
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Outside  of  new  capability  development,  Progeny  intends  to  further  mature  the  tools  developed 
under  this  SBIR  in  order  to  meet  emerging  needs  from  existing  customers. 

Additionally,  Progeny  is  investigation  commercializing  the  concepts  developed  in  this  SBIR  and 
other  SBIRs  to  provide  services  to  small  and  medium  size  businesses  that  can  not  afford  the 
expensive  assessment  batteries  offered  by  other  companies. 

1 0  Commercialization  Strategy 

Progeny  Systems  has  a  proven  track  record  in  transitioning  Phase  I  SBIRs  to  Phase  III 
commercialization.  Between  1996  and  2004  Progeny  Systems  won  39  SBIRS,  27  have  transitioned 
to  Phase  II  and  6  of  those  are  in  Phase  III.  Progeny  takes  the  approach  of  exploiting  affordable, 
mature  COTS  and  open  standard  technologies  to  provide  both  effective  and  cost-efficient  solutions. 
This  approach  also  leads  to  reliable  systems  with  low  life-cycle  costs  that  also  have  efficient  paths  for 
future  technology  refresh  without  the  restrictions  of  vendor- lock  and  proprietary  designs.  Progeny 
Systems  competencies  include  Research  and  Development,  Systems  Integration,  and  Product 
Development  and  Manufacturing.  This  unique  mix  of  capabilities  gives  us  a  competitive  advantage 
among  small  businesses  allowing  us  to  be  effective  throughout  the  entire  technology  development 
process  to  best  meet  our  customer  needs. 

10.1  Market  Need 

At  the  present  time  there  is  no  objective  means  to  assess  the  leadership  skill  levels  of  military 
officers.  This  has  lead  to  complaints  by  large  numbers  of  officers  who  have  identified  the  lack  of 
objective  assessment  as  a  serious  issue,  which  results  in  decreases  in  motivation.  The  developmental 
feedback  provided  to  them  by  superiors  which  subordmates  use  to  chart  career  development  has 
also  been  limited  due  to  the  recent  operational  tempo  of  the  military  services.  Additionally,  research 
has  shown  that  in  some  case  feedback  by  human  supervisors  is  not  as  effective  as  feedback  provided 
by  software  because  users  feel  that  software  is  less  biased  or  motivated  by  external  forces.  By  having 
software  which  is  capable  of  providing  on  demand  unbiased  skill  assessment  and  corresponding 
exercises,  officers  would  be  self-aware  of  their  skill  levels  and  more  motivated  to  perform  their  jobs. 
Further,  the  assessment  data  could  be  used  by  an  automated  development  center,  which  could 
chart/schedule  training  courses  and  or  indicate  that  the  individual  was  deserving/ prepared  for 
advancement/ promotion.  This  would  lead  to  an  increase  in  personnel  moral  and  happiness,  which 
in  turn  will  lead  to,  improved  performance.  Not  to  mention  that  the  exercises  will  continually 
increase  skill  levels. 

There  is  little  disagreement  among  military,  industry  and  academia  that  ADL  improves  costs 
and  efficiencies  by  distributing  inexpensive  instructional  components  to  physically  remote  locations 
and  the  simulation  of  expensive  devices  for  both  operator  and  maintenance  training.  The  optimal 
solution  is  to  provide  a  collaborative  environment  accessible  from  the  computer  devices  we  use 
every  day.  To  provide  the  best  accessibility,  ubiquitous  infrastructures  such  as  the  Internet,  Web  and 
communications  must  be  used.  We  anticipate  many  opportunities  for  Phase  II  prototypes  across 
DoD,  Federal,  and  civilian  organizations  that  will  position  us  for  a  Phase  III  transitional  program. 

We  intend  to  commercialize  these  components  into  our  Integrated  Learning  Environment  for 
the  NAVY,  ShipMATES,  Minimum  Bandwidth  Distributed  Simulations  for  Warfighter  Shipboard 
Training  programs,  and  other  emerging  worldwide,  network- based,  information  management 
applications.  Our  experience  has  shown  that  migrating  SBIR  developed  products  into  our  existing 
militaiy  programs  has  proven  to  be  beneficial  to  both  the  agency  responsible  for  the  research  and 
the  agency  receiving  the  developed  product. 
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10.2  Non-DoD  Market  Need 

Although  there  are  many  commercial  assessment  centers  in  existence,  the  fees  for  enrolling 
personnel  in  the  centers  are  quite  lai^e  and  cost  prohibitive  to  many  small  to  medium  companies.  By 
developing  software,  which  is  capable  of  providing  similar  assessment  performance  as  the  centers, 
the  staff  of  small  to  medium  size  companies  would  now  have  the  opportunity  to  conduct  in  depth 
skill  assessments  to  identify  areas  that  need  development.  Further  the  exercises  in  the  software 
would  provide  the  continual  means  by  which  employees  could  improve  their  skills  at  any  time. 
Improving  the  skills  of  managers  will  translate  into  higher  productivity  and  reduced  costs  for 
companies.  In  addition,  by  improving  tacit  or  interpersonal  skills,  conflicts  between  employees 
should  decrease  and  the  motivation  of  employees’  increase,  which  again  will  translate  into,  increased 
company  productivity. 
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Appendix  A:  A05-025  Simulation  Prototype  Walkthrough 


11.1  Army  Leadership  Simulation  Prototype 


Number: 

1 333333333  | 

•  111111111  -Public -3-1-1981 

Last  Name: 
i  Shonts  1 

•  222222222  -Smrth-1-1-1980 

•  333333333  -  Shonls  -  9-27-1979 

Date  Of  Birth: 

M>.!  DD  Wi-V 

|9  |27  h'1979 

1  Rarnfir  Creation  and  Pronresa  Trackino  Too!  -  Used  to  manaoe  Learners  and  their  sample  Leadership  Skills  data 

1  Login  1 

InriKidiial  1  eariershic  Simulation  Adiiistment  Toe!  -  Used  to  confiaure  simulation  parameters  for  an  individual  Learner. 

Simulation  Builder  -  Used  for  creatino  or  editina  Simulations  and  their  associated  Simulation  Adjustment  Parameters. 

The  Army  Leadership  Simulation  Prototype  for  A05-025  Phase  1  is  designed  to  present  Learners  with  challenging  leadership  situations  and 
record  the  Learners'  performance  The  prototype  currently  tracks  twelve  leadership  behaviors  and  is  easily  expandable  to  include  more  as  they 
are  defined  The  beha'/iors  are' 

•  Clarifying  Roles.  Assigning  tasks  and  explaining  job  responsibilities.  cbjectK’es  and  performance  expectations. 

•  Monitoring  operations  Checking  on  the  progress  and  quality  of  the  work,  evaluation  of  individual  and  unit  perfomiance. 

•  Short-term  Planning'  Determining  how  to  use  personnel  and  resources  to  accomplish  a  task  efficiently  and  determining  how  to 
schedule  and  coordinate  unit  activities  efficiently 

•  Consulting:  Checking  v/ith  people  before  making  decisions  that  affect  them  Encouraging  participation  in  decision  making  and  using 
ideas  and  suggestions  of  others 

•  Supporting  Acting  considerate,  showing  sympathy  and  support  when  someone  is  upset  or  anxious  and  providing  encouragement  and 
support  when  there  is  a  difficult,  stressful  task. 

«  Recognizing-  Providing  praise  and  recognition  for  effective  performance  significant  achievements  and  special  contributions  and  v 

114-1  ALS  Welcome  and  Login  Screen 

The  Army  Leadership  Simulation  Prototype  (ALS)  is  composed  of  a  simplistic  Learner  Management  System 
(LMS),  a  simplistic  Learning  Content  Management  System  (LCMS)  and  a  reference  Simulation  Engine.  Once 
a  Simulation  has  been  developed  using  the  Simulation  Builder  it  is  registered  in  the  ALS  LCMS  and  made 
available  to  all  Learners.  The  ALS  LMS  Main  Welcome  Screen  is  shown  in  Figure  114-1. 
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11.2  Taking  the  Simulation 


.*  I 


Exit  Simulation 


Current  User: 

Sut  Smith 

Clsritying  Roits:  0 
tilonitoring  Op*ritionf;  4 
Jhort-Temi  Planning:  0 
Consulting:  0 
Supporting:  t) 

Recognizing:  C 
Developing:  0 
Empo»'.»ring:  0 
Envisioning  Change:  0 
Taking  Risks  For  Change:  0 
Encouraging  Innovative 
Thinking:  0 
External  Uonitoring:  0 


Please  select  from  on  of  the  ftillov/ing  simulations; 
i  Prototype  Simulation  v 


114-2  Simulation  Selection  Screen 

After  successfully  logging  in  to  the  ALS  LMS  the  Learner  is  presented  with  the  Simulation  Selection  Screen 
(Figure  114-2).  The  drop  down  list  contains  all  of  the  Simulations  that  have  been  registered  with  the  LCMS. 
If  the  Learner  has  previously  completed  the  selected  Simulation  the  Review  button  becomes  enabled.  In  the 
left  hand  column  the  Learner’s  cumulative  Leadership  Behavior  scores  are  shown. 
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Once  the  Simulation  is  launched  the  left  column  contents  are  replaced  with  the  Simulation  Description  and 
the  current  values  of  the  simulation  variables.  This  display  is  used  for  Simulation  debugging  purposes.  In 
the  right  column  the  contents  of  the  Decision  Node  are  displayed.  These  usually  consist  of  a  Page  Title, 
Story  Text,  Story  Image  or  Media,  Question  and  possible  Decisions.  The  Learner  selects  the  appropriate 
response  from  the  drop  down  and  clicks  “Continue”. 
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Simulation  variables  are  modified  when  the  Learner  makes  a  decision.  In  this  screen  Skill03Mod  has  been 
changed  from  0  to  1.  This  indicates  that  the  Learner’s  choice  has  increased  their  Short-Term  Planning 
Leadership  Behavior.  After  the  variables  are  processed  the  Simulation  Engine  presents  the  appropriate 
destination  node  defined  by  the  decision  the  Learner  made.  While  this  demonstration  only  shows  one 
possible  path,  each  decision  made  by  the  Learner  impacts  their  path  through  the  Simulation. 
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Not  all  nodes  require  input  from  the  Learner.  In  some  cases  it  is  beneficial  to  break  up  the  Simulation  story 
into  smaller  chunks.  This  is  handled  by  the  “Continue  To”  node.  These  nodes  simply  present  the  stoiy 
content  and  a  button  that  linearly  progresses  to  the  next  node. 
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RiHfflffliisiiiiiiii 

Y,  YoYi- 

Current  User: 

Su«  Smith 

Description: 

SS  St&ri*>  of  Command: 

Conditions  for  Attack  Dissolvs 
in  Bad  W«ith»r 

Variables: 

SkiliOtMod:  [0] 

SkiiI02Mod:  [0] 

SkiimSMod;  [2] 

SkillMUod:  {0] 

SkiliOSMod:  {0] 

SkiDOiMod;  [G] 

SkillOTMod:  (0] 

Skilioeuod:  [!] 

SkillOSUod:  10] 

SkiinOMod:  {0] 

SkilMIMod:  10] 

Skill12Mod:  [0] 
iMrrtorlsTirtd:  (falsa] 
LoarnerlsPrsparad:  (true] 
LMrnarisEshausUd:  [faiss] 

:  At  0700  you  are  well  rested  and  ready  to  go  thanks  to  the  good  men  and  women 
working  for  you 

114-6  Second  Story  Progression  Node 

Figure  114-6  is  another  “Continue  To”  node. 
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Current  User: 

Smith 

Description: 

(6  Storlt*  of  Bittli  Cwnmsnd: 
Conditions  for  Attack  Dissolve 
in  Bad  Weather 

Variables: 

StullOIMod:  0 

SklllOmod:  0 

Skill93Mod:  2 

Sklll04Uod:  0 

SkillOSMod:  0 

SkilldSMod:  0 

Sklll07Mod'  0 

Skilimiod:  1 

SklllttSUod:  6 

Sklll10Uod:0 

SkliltlMod:  0 

5kill12Mod:0 

LeernerliTired:  filw 
LtarnerliPrepared:  true 
LearnerlsEshausted:  false 

Rfttiirn  Home 

Leadership  Behavior 

Clarifying  Roles. 

Monitoring  Operations. 

Short-Term  Planning 

Consulting. 

Supporting 

Adjustment  Learner  Total 

:o  :0 

:0 .  0 

;2  2 

;0  ;0 

io  0 

Recognizing 

;0 

0 

Developing: 

.0 

Empowering: 

I1 

Envisioning  Change 

0 

Taking  Risks  For  Change' 

iO 

0 

... 

Encouraging  Innovative  Thinking 

0 

:0 

V 

114-7  Simulation  End  with  Performance  Report 

Once  the  Simulation  End  has  been  reached  the  Learner  is  presented  with  a  score  card  that  shows  their 
Leadership  Behavior  adjustments  from  their  simulation  attempt  along  side  their  cumulative  Leadership 
Behavior  Scores. 
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1 1 .3  Reviewing  Your  Performance 


Exit  SimutaUon 
Current  User: 

Su»  Smith 

Clarifying  Roits:  0 
U»nit9rfng  OiMrttiont: 
Short-Term  Planning:  2 
Consulting;  0 
Supporting;  0 
Recognizing;  0 
Developing;  0 
Empotvering:  1 
Envisioning  Change;  {I 
Taking  Risks  For  Change;  0 
Encouraging  Innovative 
Thinking;  0 
Exterival  Monitoring:  0 


Please  select  from  on  of  the  following  simulations; 
Prototype  Simulation  v; 


[  Launch!  |  [  Review  j 


114-8  Simulation  Selection  Screen  (Review  Enabled) 

The  next  time  the  Learner  selects  the  Simulation  the  Review  button  is  enabled.  Reviewing  the  Simulation  lets 
the  Learner  walk  through  their  previous  actions  within  a  given  simulation. 
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Exit  Simulation 


Current  User; 


Description: 

(C  Sl&riri  of  Sittlf  Command; 
Conditions  for  Attach:  Oissohrs 
in  Bad  Wsathsr 


You  are  the  commander  of  an  armor  battalion  that  is  participating  in  a  training 
,  exercise  It  is  2300  and  your  men  are  finishing  preparations  for  the  0700  LD.  Tomorrow 
:  should  be  a  fairly  simple  day  if  things  go  according  to  plan.  The  morning  v/ill  start  out 
with  the  enemy  fortification  being  bombarded  by  aircraft  Once  their  defenses  are 
destroyed  by  air  assault  you  will  lead  the  charge  to  take  control  of  the  base.  Now 
would  be  a  good  time  for  you  to  get  sonte  rest 

What  action  would  you  like  to  take? 

■  You  Selected  Lea're  “wake-up"  cnlera  v.nth  your  command  post  and  head  to  bed 


L*4rr>*rlsTirtd:  {ftlit] 
L*«fn>flsPrtp<r»d:  ftats*] 
L»»rn»rlsEAhviust*d:  [falst] 


114-9  First  Decision  Node  with  Your  Selection 

Instead  of  being  allowed  to  make  the  decisions  the  Learner  is  shown  what  decision  they  made. 
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Current  User: 

Sim  Smith 

Description: 

SIoHm  of  Bittle  CoAimtnd: 
Conditions  for  Attack  Dissolve 
in  Bad  Weather 

Variables: 

SkillOIMod:  0 

Skiil02fldd:  0 

SkJtlOSMod:  2 

Return  Heme 

SkiliiMMod;  0 

SkiliOSUod:  0 

SkillCSMod;  0 

Leadership  Behavior 

Adjustment 

SkiliOrUod;  0 

Skill&SMod;  1 

SkillOSUod;  0 

Clarifying  Roles: 

:0 

SkiillOUod:  0 

SklllHUod;  0 

Monitoring  Operations 

iO 

5klll12Mod:  0 

LeirnerliTirid:  false 

Short-Term  Planning 

12 

LearnerlsPreparedr  true 
LearnerltEahausted:  false 

•Consulting. 

!0 

Supporting 

0 

Recognizing 

,0 

Developing 

Empowering 

:0 

1 

Envisioning  Change. 

:0 

Taking  Risks  For  Change 

0 

Encouraging  InnovalK'e  Thinking. 

:0 

V 

114-13  Review  End  with  Recorded  Simulation  Score 

At  the  end  of  a  Review  the  learner  is  presented  with  the  saved  score  card.  Unlike  the  score  card  at  the  end  of 
taking  a  simulation,  the  review  mode  just  shows  the  Leadership  Behavior  score  adjustments  achieved  from 
the  reviewed  simulation  attempt. 
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1 1 .4  Effect  of  Individual  Adjustment  Parameters 


114-14 ILSAT  Tool  Modifying  the  "Needs  Sleep"  Setting 

The  Simulation  Engine  has  a  method  for  allowing  Training  Administrators  to  adjust  settings  for  individual 
Learners.  The  available  settings  are  determined  by  the  Simulation  authors.  Figure  114-14  shows  the  Training 
Administrator  setting  the  “Needs  Sleep”  parameter  to  “true”. 
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Current  User; 

Joi  Public 

Description: 

(C  Stdriei  of  Sittlc  Comnund; 
Conditions  for  Attack  Dissolve 
in  Bad  Weather 


LeernerlsTired:  {false} 
LearnerlfPrepared:  {false] 
LearnerlsEshausted:  [true] 


}  You  are  the  commander  of  an  armor  battalion  that  is  participating  in  a  training 
1  exercise.  It  is  2300  and  your  men  are  finishing  preparations  for  the  0700  LD.  Tomorrow 
j,  should  be  a  feirly  simple  day  if  things  go  according  to  plan.  The  morning  will  start  out 
'  wth  the  enemy  fortification  being  bombarded  by  aircraft  Once  their  defenses  are 
,  destroyed  by  air  assault  you  will  lead  the  charge  to  take  control  of  the  base.  Now 
<  would  be  a  good  time  for  you  to  get  some  rest 

1  What  action  would  you  like  to  take? 

!  i  Leave  “wake-up"  crilera  with  your  command  post  and  head  to  bed 


[Leave  “wakeHiD"  critera  with  vour  command  c 

^ - 


)3t  and  head  to  bed. 


114-15  Learner  with  Limited  Options 

The  Learner  taking  the  Simulation  in  Figure  1 14- 15  is  limited  to  two  options. 
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Current  User: 

John  Shonti 

Description: 

i6  St»ri*s  of  BitU*  Command: 
Conditions  for  Attack  DIssotva 
in  Bad  Waathar 


LaarnarltTirtd:  (falsa] 
LaarnerliPraparad:  (falst] 
LaarnerlsEahaustcd:  (falsa] 


;  You  are  the  commander  of  an  armor  battalion  that  is  participating  in  a  training 
i  exercise  It  is  2300  and  your  men  are  finishing  preparations  for  the  0700  LD  Tomorrow 
;  should  be  a  fairly  simple  day  if  things  go  according  to  plan.  The  morning  will  start  out 
;  with  the  enemy  fortification  being  bombarded  by  aircraft  Once  their  defenses  are 
I  destroyed  by  air  assault  you  will  lead  the  charge  to  take  control  of  the  base.  Now 
i  would  be  a  good  time  for  you  to  get  some  rest. 

‘  What  action  would  you  like  to  take? 


Leave  "wake-up”  critera  with  your  command  post  and  head  to  bed 


Stay  awake  and  personalty  oversee  all  of  the  preparations  for  tomorrow's  excercise 
Rush  off  to  bed _ 


114-16  Learner  with  all  available  options 

The  Learner  taking  the  Simulation  in  Figure  114-16  has  all  three  options  available. 
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12  Appendix  B:  Evaluation  Metrics  for  Gaming  Engines 

The  selected  engines  were  evaluated  based  on  their  ability  to  provide  the  following  functionality 

•  Ability  to  deliver  compelling  visual  representations  of  simulation  objects 

•  Ability  to  deliver  audio  to  support  the  simulation 

•  Ability  to  be  deployed  over  the  web 

•  Ability  to  script  actions  and  objects  within  the  simulation  environment 

•  The  functionality  measurements  have  been  defined  as  Excellent,  Good,  Poor  and  None. 

•  Additionally,  the  engines  were  evaluated  on  the  following  criteria: 

•  Initial  Cost 

•  Licensing  Terms 

•  Target  System  Requirements  (Hardware  and  Software)  in  terms  of  Cost 

•  Ease  of  Development  in  terms  of  Cost 

•  Ease  of  Maintenance  in  terms  of  Cost 

The  cost  measurements  have  been  defined  as  Expensive,  Affordable,  Cheap,  and  Free, 

Licensing  Terms  have  been  defined  as  Difficult  to  Comply,  Simple  to  Comply,  or  No 
Compliance  Required.  These  are  marked  in  the  Evaluation  Matrix  as  Difficult,  Simple  and  None 
respectively. 

Since  Ease  of  Development  and  Maintenance  vary  dependmg  on  the  developer’s  skill  the 
evaluation  assumes  that  the  developer  has  no  knowledge  of  the  specific  programming  language  or 
engine.  Therefore,  “ease”  is  based  on  the  relative  learning  curve  for  each  development 
environment. 

12.1  Evaluated  Engines 

The  following  engines  were  evaluated  using  the  above  Evaluation  Metrics  to  determine  which 
engine  would  be  most  suitable  for  developing  the  prototype  required  by  the  SBIK  Progeny’s  end 
goal  is  to  provide  interfaces  to  the  prescriptive  learning  and  simulation  control  components  that  will 
work  in  almost  any  engine.  For  the  purpose  of  this  evaluation  it  is  important  to  note  that  ES3  and 
Unreal  have  an  “unfair”  advantage  over  the  other  engines  due  to  their  existing  use  by  the  Army. 
This  evaluation  has  attempted  to  not  include  any  benefits  provided  by  this  existing  use.  However, 
these  benefits  are  worth  looking  into  for  work  beyond  the  Phase  I  prototype. 


Visual 

Appearance 

Functionality 

Audio  Scripting 

Support  Support 

Web 

Deployment 

Initial  Cost 

System 

Reqs. 

Costs 

Development  Maintenance 

License 

Terms 

Cube 

Good 

Good 

Good 

Good 

Free 

Affordable 

Cheap 

Cheap 

Simple 

M 

OgreSd 

Excellent 

None 

None 

None 

Free 

Affordable 

Expensive 

Expensive 

Simple 

C 

UGL 

Good 

Good 

None 

None 

Free 

Affordable 

Expensive 

Expensive 

Simple 

C 

Source 

Excellent 

Excellent 

Excellent 

Poor 

Expensive 

Expensive 

Expensive 

Expensive 

None 

•o 

Unreal  (PC) 

Excellent 

Excellent 
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12.2  Cube 

Vendor:  Open  Source  (creaton  Wouter  van  Oortmerssen) 

License:  ZLIB 
Cost:  Free 

URL:  http:/ / www.cubeengme.com/ 

12.2.1  Strengths 

In- engine  editing  environment 

Easy  to  develop  simple  environments 

Contains  existing  assets 

Community  support  with  available  free  assets 

1 2.2.2Weaknesses 

Visuals  are  sub  standard 

Environments  lack  any  curvature  and  end  up  looking  very  blocky  this  severely  detracts  from  the 
experience 

Included  assets  don’t  apply  to  this  effort 

12.2.3Sunimaiy 

The  Cube  engine  is  an  Open  Source  First  Person  Shooter  creation  tool.  It’s  most  innovative 
feature  is  the  in-engine  editor.  This  editor  allows  the  simulation  developer  to  virtually  walk  around 
the  environment  and  add  objects  to  their  current  location.  Cube  also  has  a  decent  community 
following  that  provides  assets  and  support  to  other  developers.  The  ease  of  use  provided  by  this 
engine  is  counter  balanced  by  the  limited  geometry  support  within  the  environment.  No  matter 
how  detailed  the  environment  textures,  simulations  always  appear  to  take  place  in  a  3D  version  of 
Tetris. 

The  successor  to  Cube,  Sauerbraten  (http://strlen.com/sauerbraten/index.htm^  appears  to 
have  overcome  the  geometry  limitations.  At  the  time  of  the  evaluation  this  engine  is  not  fully 
functional.  In  the  effort  beyond  Phase  I  this  engine  should  be  re-examined. 

12.3  Ogre3d 

Vendor:  Open  Source 

License:  LGPL 
Cost:  Free 

URL:  http:/ / www.ogre3d.oig 

12.3.1  Strengths 

Provides  easy  access  to  powerful  3D  rendering  technologies. 

1 2.3.2Weaknesses 

Just  a  3D  rendering  Hbrarynot  a  full  game  /  simulation  engine. 

Requires  C-t-i-  development  skills. 
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12.3.3Summary 

Ogre3D  is  a  veiy  powerful  and  well  documented  3D  rendering  library.  It  has  been  used  by 
several  projects  to  handle  the  visual  processing  for  various  games  and  other  non-game  related 
applications.  While  it  can  be  utilized  to  provide  very  vivid  and  rich  3D  visuals  to  a  simulation  it 
doesn’t  bring  the  full  package.  Libraries  for  AI,  sound,  control,  scripting  and  networking  would  all 
need  to  be  sourced  or  developed.  If  Progeny  intended  to  develop  a  custom  3D  simulation 
environment  Ogre3D  would  likely  be  used  to  abstract  the  visual  component. 

12.4  Lightweight  Java  Gaming  Library 

Vendor:  Open  Source 

License:  BSD 
Cost:  Free 

URL:  http:/ /Iwjgl.oi^ 

12.4.1  Strengths 

Utilizes  Java  which  enables  it  to  easily  run  on  multiple  target  environments  with  little  to  no 
modification. 

Easily  exposes  Graphics,  Audio  and  Control  components  to  the  Java  developer. 

1 2.4.2  Weaknesses 

Not  a  full  game  /  simulation  engine. 

Requires  Java  development  skills. 

12.4.3Summary 

The  Lightweight  Java  Gaming  Library  (LJGL)  is  a  set  of  APIs  that  allow  the  Java  developer  to 
utilize  OpenGL,  OpenAL  and  various  Game  Pads  and  other  input  devices  while  developing 
simulations  or  games.  While  the  LJGL  provides  more  components  than  Ogre3D  it  is  simply  a 
library  to  aid  in  simulation  /  game  development.  A  skilled  Java  developer  is  still  required  to 
implement  AI,  networking  and  scripting  elements. 

12.5  Source 

Vendor:  Valve 

License:  Proprietary 
Cost:  Disclosed  under  NDA 

URL:  http:/ / www.valvesoftware.com/business.html 

12.5.1  Strengths 

Powerful  graphics  engine  that  supports  DirectX6  through  DirectX9  hardware 

Powerful  scripting  engine 

Powerful  AI  engine 

Powerful  Physics  engine 

Vehicle  support 

Lai^e  community  support 

1 2.5.2Weaknesses 

Expensive 
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Developers  must  be  familiar  with  using  the  provided  development  tools 
Not  designed  for  rapid  simulation  development 

Due  to  high  quality  assets  final  package  size  is  lai^e  making  web  deployment  an  issue 

12.5.3Summary 

The  Source  engine  is  the  same  engine  that  powers  Half-Life  2  and  Counter  Strike:  Source. 
These  are  two  of  the  most  popular  games  played  online.  Due  to  it’s  popularity  eveiy  feature  of  the 
engine  has  been  extensively  tested  by  the  developers  and  the  players.  Many  third  party  “Mod” 
makers  have  developed  assets  that  can  be  utilized  by  the  Source  engine.  Some  of  these  assets 
pertain  to  the  SBIR  effort  (weapons,  vehicles  etc)  and  can  be  utilized  for  free.  The  AI  and  scripting 
abilities  are  phenomenal  and  compare  to  the  Unreal  engines  capabilities.  The  Source  engine’s 
realistic  physics  and  character  animation  features  go  a  long  way  in  creating  life  like  environments. 
With  all  of  these  features  comes  a  hefty  price  both  in  upfront  cost  and  in  development  time. 
Simulations  developed  using  the  Source  engine  are  lil^ly  to  take  many  months  to  a  few  years  each  to 
complete.  Another  issue  is  the  file  size  of  a  completed  simulation.  By  utilizing  high  quality  models 
and  textures  the  simulation  would  be  difficult  to  deploy  over  the  web. 

12.6  Unreal  2  (PC) 

Vendor:  Epic  Games 

License:  Proprietary 

Cost:  Disclosed  under  NDA 

URL:  http:/ / www.unrealtechnology.com 

12.6.1  Strengths 

Powerful  graphics  engine 

Powerful  scripting  engine 
Powerful  AI  engine 
Powerful  Physics  engine 
Vehicle  support 
Lai^e  community  support 

Currently  Used  by  Army  for  the  America’s  Army  simulation 

1 2.6.2  Weaknesses 

Expensive 

Developers  must  be  familiar  with  using  the  provided  development  tools 
Not  designed  for  rapid  simulation  development 

Due  to  high  quality  assets  final  package  size  is  lai^e  making  web  deployment  an  issue 

1 2.6.3Summary 

The  Unreal  2  engine  has  been  used  in  many  award  winning  games.  It  has  incredibly  powerful 
features  and  a  lai^e  community  of  “Mod”  creators.  Some  assets  in  the  “Mod”  community  can  be 
used  by  this  SBIR  at  no  cost.  The  US  Army  is  already  using  the  Unreal  2  Engine  for  the  America’s 
Army  simulation  (http://www.americasarmy.com)  which  contains  Army  approved  assets.  These 
assets  may  be  usable  by  this  SBIR  effort.  Like  the  Source  engine,  all  the  power  provided  by  the 
Unreal  engine  comes  at  a  great  price.  While  the  upfront  cost  is  not  disclosed  the  Unreal  2  (^Gjox) 


45 


pricing  should  provide  a  hint.  Epic  Games  also  has  non-traditional  licensing  that  they  can  offer 
which  America’s  Army  utilized.  Further  research  may  reveal  that  the  Unreal  Engine  could  be  very 
affordable.  The  other  major  drawback  is  the  resulting  simulation  file  size.  The  large  size  will  make 
web  based  deployment  difficult. 

1 2.7  Unreal  2  (Xbox) 

Vendor:  Epic  Games 

License:  Proprietary 

Cost:  $350,000  +  3%  of  revenue  from  game 
URL:  http:/ / www.unrealtechnology.com 

12.7.1  Strengths 

Powerful  graphics  engine  that  supports  DirectX6  through  DirectX9  hardware 

Powerful  scripting  engine 

Powerful  AI  engine 

Powerful  Physics  engine 

Vehicle  support 

Inexpensive  deployment  platform 

1 2.7.2Weaknesses 

Expensive  software 

Xbox  development  hardware  is  expensive 

Non- web  deployable 

Difficult  to  update  even  using  Xbox  Live! 

Difficult  to  import  skill  sets 
Difficult  to  export  learner  progress 
12.7.3Suniniary 

After  meeting  with  ICT  and  learning  about  the  success  of  Full  Spectmm  Warrior  and  their 
reasoning  for  choosing  the  Xbox  as  a  deployment  platform,  Progeny  decided  to  investigate  the 
technologies  that  ICT  used.  It  became  immediately  apparent  that  this  technology  would  not  meet 
our  needs  due  to  the  deployment  platform’s  limitation  with  exporting  learner  progress.  While  the 
Xbox  provides  a  very  stable  and  affordable  deployment  platform  it  is  far  too  expensive  and  difficult 
to  develop  simulations  for. 

1 2.8  Every  Soldier  a  Sensor  Simulation 

Vendor:  US  Army  &  ICT 

License:  N/A 

Cost:  Already  Owned  By  Army 

URL:  https:/ / slimes3.rdecom.army  (Requires  AKO  account) 

12.8.1  Strengths 

Easily  deployable 

Can  easily  add  assets  to  the  simulation 
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Can  easily  create  new  simulations  with  the  built-in  editor 
Designed  with  Army  training  in  mind 
Is  a  complete  simulation  development  package 
Coding  is  not  required 

1 2.8.2Weaknesses 

Poor  visuals 

Most  assets  are  2D  bitmaps  which  always  face  the  user  in  the  3D  environment 
Limited  built-in  maps 
Not  Scriptable 

12.8.3Summary 

Every  Soldier  a  Sensor  Simulation  (ES3)  is  a  training  tool  developed  by  the  ICT  and  is  currently 
in  use  by  the  Army.  It  was  developed  specifically  for  Army  training,  easy  web  deployment,  and  easy 
simulation  creation.  Adding  new  assets  to  the  simulation  development  tool  is  trivial  and  can  be 
done  using  slightly  modified  digital  camera  images.  This  allows  content  creators  to  simply  take  a 
photo  of  an  object,  scale  it  down  and  remove  the  background  then  import  it  into  the  simulation. 
Content  creators  could  develop  situationally  pertinent  training  while  deployed  in  the  field.  Since 
ES3  was  designed  with  a  specific  Army  training  methodology  in  mind  it  does  not  support  the  level 
of  scripting  that  is  required  for  this  SBIR.  Progeny  will  need  to  work  with  ICT  to  add  this 
functionality.  Additionally,  a  process  for  easily  adding  new  maps  would  need  to  be  developed. 

12.9  Flash  Professional 

Vendor:  Macromedia 

License:  None 
Cost:  $699 

URL:  http:/ / www.macromedia.com/ software/ flash/flashpro 

12.9.1  Strengths 

Powerful  rendering  engine 

Powerful  scripting  engine 
Powerful  animation  tools 
Extensive  community  support 
Large  number  of  free  assets 
Small  resulting  file  sizes 
Easily  deployed 

Runs  on  multiple  platforms  and  devices 

Can  utilize  a  variety  of  media  formats  including  MP3,  QuickTime  Movies  and  Digital  Images 

1 2.9.2Weaknesses 

Not  specifically  a  simulation  engine 

Not  specifically  a  3D  engine 
Requires  Flash  development  skills 
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12.9.3Summaiy 

Macromedia  Flash  is  practically  a  ubiquitous  standard  for  deploying  rich  media  on  the  web. 
The  Flash  player  has  been  included  as  part  of  the  Microsoft  Windows  install  and  is  standard  on 
almost  all  PCs.  The  Flash  player  is  supported  on  all  popular  operating  systems  and  many  mobile 
devices  such  as  Phones  and  PDAs.  Simple  Flash  development  is  easy  and  compelling  “movies”  can 
be  created  in  a  short  amount  of  time.  Advanced  Flash  development  simply  combines  the  “movies” 
with  some  Action  Script.  The  Advanced  development  is  more  rapid  than  other  environments  such 
as  Source  and  Unreal.  Even  though  Flash  isn’t  designed  specifically  for  creating  simulations  a 
reusable  simulation  engine  can  be  easily  developed.  Progeny  would  develop  this  simulation  engine 
for  the  SBIR  prototype.  One  of  the  pivotal  features  is  Flash’s  web  deployment  capability.  The 
technology  was  designed  specifically  for  delivering  rich  content  over  the  web.  The  small  SWF  files 
generated  by  the  Flash  development  environment  are  directly  supported  by  popular  Learning 
Content  Management  Systems  (LCMS)  such  as  OutStart’s  Evolution. 

12.10FPS  Creator 

Vendor:  The  Game  Creators 

License:  No  Royalty 
Cost:  $50.00 

URL:  http:/ / www.fpscreator.com 

1 2.1 0.1  Strengths 

Rapid  Development  (a  simple  game  using  default  models  and  objects  can  be  made  ia  as  little  as  15 
minutes) 

Low  Cost 

Included  assets  with  a  user  community  that  has  developed  more  free  assets 

Engine,  assets  and  resulting  works  are  all  royalty  free.  Games  created  can  be  given  away  or  sold 
with  no  obligation  to  compensate  the  vendor  beyond  the  original  purchase  price. 

The  engine  source  code  is  available  for  modification. 

Source  code  does  not  need  to  be  released  even  if  the  engine  is  modified. 

Competent  default  AI  with  full  waypoint  and  scripting  support. 

1 2.1 0.2  Weaknesses 

Default  assets  are  WW2  and  Sci-Fi  which  don’t  apply  to  this  effort. 

Engine  is  “indoor”  which  prevents  expansive  terrain  development  seen  in  Unreal  or  Source 

12.1 0.3Sumniary 

FPS  Creator  is  First  Person  Shooter  rapid  development  tool.  It  contains  an  intuitive  editor,  pre¬ 
built  room  models  and  a  long  list  of  other  assets.  Using  the  included  assets  a  fully  fimctional  game 
can  be  created  in  minutes.  Tbe  Engine,  Models  and  other  assets  are  all  royalty  free  and  can  be  sold 
or  given  away  with  no  mention  of  FPS  Creator  or  The  Game  Creators  included. 


12.11  Results  of  Simulation  research 

The  results  for  the  Phase  I  prototype  effort  pointed  to  the  use  of  Macromedia  Flash.  While  not 
as  easy  to  develop  simulations  with  initially  as  some  of  the  other  engines  Progeny  intended  to  release 
the  code  that  powers  the  prototype  simulation.  This  code  can  be  utilized  by  future  simulations  to 
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reduce  development  time.  The  primary  reason  why  Progeny  chose  to  utilize  Flash  for  the  Phase  I 
prototype  is  its  ability  to  be  deployed  over  the  web.  Progeny  will  not  have  to  develop  extensive 
code  to  deliver  Flash  simulations  through  existing  LCMS  software  such  as  OutStart’s  Evolution. 
This  will  greatly  reduce  the  work  needed  to  build  an  effective  training  simulation  prototype.  Since 
the  gaming  development  aspect  of  the  phase  I  effort  was  deprecated,  the  analysis  as  to  the  preferred 
gaming  engine  will  be  revisited  if  Progeny  is  awarded  a  Phase  II. 
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